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Chapter 1. 
Overview of PISA 2009 



1.1 Introduction 

The Program for International Student Assessment (PISA) is an international assessment of 15-year-old 
students. PISA measures how well students apply their knowledge and skills learned in and out of 
school in reading, mathematics, and science literacy to problems within a real-life context. PISA also 
gathers information from students and school principals about school context and from students about 
their educational experiences and attitudes. Analyses of PISA data have provided information about the 
relative performance of students across nations, as well as variation across nations in relationships 
between student background, attitudes, and experiences and their performance in mathematics, reading, 
and science literacy at age 15. 

PISA, which began in 2000, is conducted in 3-year cycles. Each PISA administration assesses one of the 
three subject areas in depth, although all three are assessed in each cycle. Assessing all three areas allows 
participating countries to have an ongoing source of achievement data in every subject area while 
rotating one area as the main focus over the years. In the fourth cycle of PISA, reading was the subject 
area assessed in depth, as it was in 2000 (see figure 1). PISA 2009 included an in-depth assessment of 
reading, as well as shorter assessments of mathematics and science. The in-depth assessment of reading 
included a sufficient number of items to develop performance estimates for reading literacy overall as 
well as for subscales involving specific aspects or processes of reading: accessing or retrieving 
information, integrating and interpreting, and reflecting and evaluating. Subscale scores were not 
available for mathematics literacy or science literacy for 2009 (subscale scores are available for 
mathematics in the 2003 PISA data and for science in the 2006 data). In addition, PISA 2009 collected 
information on students’ backgrounds, attitudes towards reading, and learning strategies. School 
principals also were asked to provide information on school demographic characteristics and the learning 
environment in the school. 



Figure 1. PISA administration cycle 



Assessment 

year 


2000 


2003 


2006 


2009 


2012 


2015 


Subjects 

assessed 


READING 

Mathematics 

Science 


Reading 

MATHEMATICS 

Science 

Problem solving 


Reading 

Mathematics 

SCIENCE 


READING 

Mathematics 

Science 


Reading 

MATHEMATICS 

Science 

Problem solving 


Reading 

Mathematics 

SCIENCE 



NOTE: Reading, mathematics, and science literacy are all assessed in each assessment cycle of the Program for International Student Assessment (PIS A). A 
separate problem-solving assessment was administered in 2003 and is planned for 2012. The subject in all capital letters is the major subject area for that cycle. 
SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 



The PISA 2009 main study in the United States consisted of three major elements: (1) a 2-hour student 
assessment of reading literacy, mathematics literacy, and science literacy; (2) a student questionnaire that 
required approximately 30 minutes to complete; and (3) a school questionnaire to be completed by the 
principal or designee that also required approximately 30 minutes to complete. 
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1.2 What PISA Measures 

PISA assesses the application of knowledge in reading, mathematics, and science literacy to problems 
within a real-life context (OECD 1999). PISA uses the term “literacy” in each subject area to denote its 
broad focus on the application of knowledge and skills. For example, when assessing reading, PISA 
assesses how well 15-year-old students can understand, use, and reflect on written text for a variety of 
purposes and settings. In science PISA assesses how well students can apply scientific knowledge and 
skills to a range of different situations they may encounter in their lives. In mathematics, PISA assesses 
how well students analyze, reason, and interpret mathematical problems in a variety of situations. Scores 
on the PISA scales represent skill levels along a continuum of literacy skills. PISA provides ranges of 
proficiency levels associated with scores which describe what a student can typically do at each level 
(OECD 2006). 

PISA’s target age of 15 allows jurisdictions to compare outcomes of learning as students near the end of 
compulsory schooling. PISA’s goal is to answer the question, “what knowledge and skills do students 
have at age 15?” taking into account schooling and other factors that may influence their performance. 

In this way, PISA’s achievement scores represent a “yield” of learning at age 15, rather than a direct 
measure of attained curriculum knowledge at a particular grade level. Fifteen-year-old students 
participating in PISA from the United States and other jurisdictions are drawn from a range of grade 
levels. In PISA 2009, 69 percent of the U.S. students were enrolled in grade 10, and another 20 percent 
were enrolled in grade 1 1 . 

To provide valid estimates of student achievement and characteristics, PISA selects a sample of students 
that represents the full population of 15-year-old students in each participating jurisdiction. This 
population is defined internationally as 15-year-olds attending both publicly and privately controlled 
schools in grade 7 and higher. Students in all programs of study (e.g., academic, vocational) are included 
in PISA. A minimum of 4,500 students from a minimum of 150 schools was required in each 
jurisdiction. In PISA 2009, jurisdictions were permitted to exclude up to 5 percent of the population of 
15-year-old students by excluding students who were unable to participate in the assessment 
(e.g., because of functional or intellectual disabilities, or limited language ability in the test language) or 
who were attending schools that were inaccessible to the test administrators. All but 5 countries 
(Denmark, Luxembourg, Canada, Norway and the United States) achieved this standard, and in 36 
countries and jurisdictions, the overall exclusion rate was less than 2 percent. When language exclusions 
were accounted for (i.e., removed from the overall exclusion rate), the United States no longer had an 
exclusion rate greater than 5 percent. (For more information on allowed exclusions see chapter 2; for 
additional details on coverage of student populations, see OECD (2010a)). 

In addition to PISA, the United States has for many years conducted assessments of student 
achievement at a variety of grade levels and in a variety of subject areas through the National Assessment 
of Educational Progress (NAEP), Trends in International Mathematics and Science Study (TIMSS), and 
Progress in International Reading Literacy Study (PIRLS). These studies differ from PISA in terms of 
their purpose and design (see appendix A). NAEP reports information on achievement of U.S. students 
using benchmarks of performance (i.e., basic, proficient, advanced achievement levels) established 
through the collaborative input of a wide range of experts and participants from government, education, 
business, and public sectors in the United States. Furthermore, the information is used to monitor 
trends in achievement specific to U.S. students and at the state level. 

To provide a critical external perspective on the mathematics, science, and reading achievement of U.S. 
students, in addition to PISA, the United States participates in TIMSS and PIRLS. TIMSS provides the 
United States with information on the mathematics and science achievement of 4th- and 8th-grade 
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students compared to students in other countries. PIRLS allows the United States to make international 
comparisons of the reading achievement of students in the fourth grade. TIMSS and PIRLS seek to 
measure students’ mastery of specific knowledge, skills, and concepts and are designed to broadly reflect 
curricula in the United States and other participating countries, in contrast to PISA which does not focus 
explicitly on curricular outcomes but rather on the application of knowledge to problems in a real-life 
context. 

1.3 PISA 2009 Administration 

PISA is sponsored internationally by the Organization for Economic Cooperation and Development 
(OECD). PISA 2009 was coordinated and administered internationally by the PISA international 
consortium, led by the Australian Council for Educational Research (ACER), through a contract with the 
OECD. Technical standards and a series of manuals provided standardized procedures for all countries 
to follow. Sixty-five countries and other education systems located in non-national entities participated. 
More than 400,000 students participated worldwide. 

The National Center for Education Statistics (NCES) was responsible for the implementation of PISA 
in the United States in accordance with the international standards and procedures. PISA 2009 data 
collection and associated tasks were carried out through a contract with Windwalker Corporation and its 
two subcontractors, Westat and Pearson. Windwalker Corporation was responsible for project 
coordination, preparation of recruitment materials, adaptation of the international instruments, 
preparation of the U.S. data files, and reporting. Westat was responsible for school and student sampling, 
recruitment of schools and students, and data collection. Pearson was responsible for the printing of 
materials, data entry, and coding and scoring. The key personnel involved in data collection included a 
school coordinator (a school staff member designated by the principal), and a test administrator and 
assistant administrator (both Westat employees). In 2009, 5,233 U.S. students and 165 U.S. schools 
participated. Data collection occurred from September 23, through November 19, 2009, and the final 
report and data were released on December 7, 2010. 

1.4 Organization of This Document 

This technical report and user’s guide is designed to provide researchers with an overview of the design 
and implementation of the 2009 Program for International Student Assessment (PISA), as well as with 
information on how to access the PISA 2009 data. This information is meant to supplement that 
presented in Organization for Economic Cooperation and Development (OECD) publications by 
describing those aspects of PISA 2009 that are unique to the United States. Chapter 2 provides 
information about sampling requirements and sampling in the United States. Chapter 3 describes 
participation rates at the school and student level. Chapter 4 describes the details of how schools and 
students were recruited, while Chapter 5 provides information on instrument development. Chapter 6 
describes field operations used for collecting data, while Chapter 7 provides detail concerning various 
aspects of data management. Chapter 8 describes international activities related to data processing, 
scaling, and weighting. Chapter 9 describes the data available from both international and U.S. sources, 
while Chapter 10 discusses some special issues involved in analyzing the PISA 2009 data. 
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The PISA 2009 U.S. sample for the main study was selected using a two-stage design — a sample of 
schools and a sample of students within sampled schools. The two-stage sample design was 
implemented to attain an approximately self- weigh ting sample of students where each 15-year-old 
student in the United States had an equal probability of being selected for the study. 

2.1 International Requirements 

To provide valid estimates of student achievement and characteristics, the sample of PISA students had 
to be selected in a way that represented the full population of 15-year-old students in each jurisdiction. 
The international desired population in each jurisdiction consisted of 15-year-olds attending both 
publicly and privately controlled schools in grade 7 and higher. A minimum of 4,500 students from a 
minimum of 150 schools was required in each jurisdiction. 

The international guidelines specified that within schools, a sample of 35 students was to be selected in 
an equal probability sample unless fewer than 35 students age 15 were available (in which case all 
students were selected). International standards required that students in the sample be 15 years and 3 
months to 16 years and 2 months at the beginning of the testing period. In the United States, sampled 
students were born betweenjuly 1, 1993 and June 30, 1994. 

The school response rate target was 85 percent for all jurisdictions. A minimum of 65 percent of schools 
from the original sample of schools were required to participate for a jurisdiction’s data to be included in 
the international database, jurisdictions were allowed to use replacement schools (selected during the 
sampling process) to increase the response rate once the 65 percent benchmark had been reached. 

PISA 2009 also required a minimum participation rate of 80 percent of sampled students from schools 
within each jurisdiction. A student was considered to be a participant if he or she participated in the first 
testing session or a follow-up or makeup testing session. Data from jurisdictions not meeting this 
requirement could be excluded from international reports. 

PISA’s intent was to be as inclusive as possible. Guidelines allowed for schools to be excluded for 
approved reasons (for example, remote regions, very small schools, or special education schools). 
Schools used the following international guidelines on student exclusions: 

Students with functional disabilities. These were students with a moderate to severe permanent 
physical disability such that they could not perform in the PISA testing environment. 

Students with intellectual disabilities. These were students with a mental or emotional disability and 
who had been tested as cognitively delayed or who were considered in the professional opinion of 
qualified staff to be cognitively delayed such that they could not perform in the PISA testing 
environment. 

Students with insufficient language experience. These were students who met the three criteria of 
not being native speakers in the assessment language, having limited proficiency in the assessment 
language, and receiving less than one year of instruction in the assessment language. 
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Overall estimated exclusions including both school and student exclusions were to be under 5 percent of 
the PISA target population. 

2.2 School Sampling in the United States 

The 2009 PISA school sample was drawn for the United States in July 2008. The sample design for this 
school sample was developed to retain most of the properties of the PISA 2006 U.S. school sample, and 
to follow international requirements as given in the School Sampling Preparation Manual: PISA 2009 
Main Study (March 2008). The school universe includes all educational institutions that serve PISA 
eligible students at age 15. 

The U.S. school sampling frame was developed from two national databases in the National Center for 
Education Statistics — public schools in the Common Core of Data (CCD, http:/ /nces. ed.gov/ccd/) and 
private schools in the Private School Survey (PSS, http:/ /nces. ed.gov/ surveys/pss/). These sources 
provide full coverage of all PISA-eligible students in the education system in the U.S. The PISA 2009 
school frame was constructed using the 2005-2006 CCD and the 2005-2006 PSS, the most current data 
at the time of the PISA frame construction. 

Eligible schools in the PISA 2009 school frame included 67,309 schools. These included schools with 
grade 7 or higher operating in the 50 states and the District of Columbia, ungraded schools, Department 
of Defense (DoD) domestic schools, Bureau of Indian Education (BIE) schools, special education 
schools, and vocational education schools. Schools in Puerto Rico and U.S. territories, DoD schools 
overseas, adult education institutions with no PISA eligible students and non-education institutions (e.g., 
home bound schools, correspondence schools) were ineligible for the study. 

A small fraction of PISA eligible schools were excluded in the United States because administration of 
the PISA assessment within these schools would not be feasible. The excluded schools were: special 
education schools for students with physical disabilities, and schools in hospitals, training centers, and 
detention centers. A total of 1,251 schools were excluded from sampling. The student loss as a result of 
these exclusions was estimated at 15,199 students or 0.36 percent of the enrollment of 15-year-old 
students. 

Schools were stratified by Census region (Northeast, Midwest, South, and West) and school type (public 
and private). Within each stratum, schools were sorted by grade range, locality, first three digits of the 
zip code, high/low minority percentage, and student enrollment. A systematic sample was selected 
independently in each stratum. The selection probability for each school was proportional to a measure 
of size based on the target cluster size (TCS) of 42 students in each sampled school. The student 
population for the 2009 PISA is defined by age to include all students between 1 5 years and 3 
(completed) months and 16 years and 2 (completed) months at the start of the testing period. The 
enrollment of PISA-eligible students (ENR) in each school was estimated using school enrollment and 
the percentage of age-eligible students per grade. 

The U.S. school sample included 236 schools — 201 large schools with at least 42 estimated eligible 
students, 10 moderately small schools with between 21 and 45 estimated eligible students, and 25 very 
small schools with less than 21 estimated eligible students. Table 1 shows some other characteristics of 
schools selected to be in the sample. 
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Table 1. Characteristics of sampled schools 



Characteristic 


Number of schools 


Type of school 


Public 


209 


Private 


27 


Geographic type 


City 


72 


Urban area 


80 


Town 


28 


Rural 


56 


Census region 


Northeast 


44 


Midwest 


54 


South 


81 


West 


57 


Grade range 


Middle/junior high school 


11 


High school 


191 


Other (K-8, 6-12, etc.) 


25 


Total school enrollment 


Less than 500 


54 


500-999 


45 


1,000-1,999 


77 


2,000 or more 


60 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International 
Student Assessment (PISA), 2009. 

In addition, for each school selected in the sample, the schools direcdy above and below the sampled 
school in the sampling frame were designated as substitute schools. The first school following the 
sample school was the first substitute and the first school preceding it was the second substitute. If an 
original school refused to participate, the first substitute was then contacted. If that school also refused 
to participate, the second substitute was then contacted. 

There were several constraints on the assignment of substitutes. A sampled school was not allowed to 
be a substitute for another, and a given school could not be assigned to be a substitute for more than 
one sampled school. Furthermore, substitutes were required to be in the same explicit stratum as the 
sampled school. If the sampled school was the first or last school in the stratum, then the second school 
following or preceding the sampled school was identified as the substitute. If the first substitute school 
did not have the same implicit stratification values as the sampled school, the first and second substitute 
schools were switched. Under these rules, it was possible to identify two substitutes for each sampled 
school. 

The PISA 2009 school sample was also selected to minimize potential overlaps with the school sample 
for the High School Longitudinal Survey (HSLS), an U.S. education study with data collection beginning 
in fall 2009. If any PISA substitute school overlapped with an originally sampled or first substitute 
HSLS school, the substitute was not to be contacted for PISA. Under this rule, 36 such schools were 
eliminated from the list of PISA substitute schools. 

2.3 Student Sampling 

In order to achieve the required student yield of 35 assessed students per school (taking into account 
student exclusions and absences), the United States set a target cluster size (TCS) of 42 students per 
school. The TCS for the main study was similar to the TCS used on PISA 2006 and was approved by 
the international consortium. 
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School coordinators were asked to provide lists of all 15-year-old students (defined as students with 
birthdates between July 1, 1993 and June 30, 1994) in their schools. To reduce burden, a simple form 
was provided to schools both in hard copy and as an attachment in the follow-up email to the first 
mailing to school coordinators (exhibit 1). 

A total of 56,221 students were listed from the 165 participating schools. The average list size was 341 
students. Once the list of students was received from a school, it was formatted for importing into 
KeyQuest, the sampling and data management software provided by ACER. 

After importing the list from a school, the appropriate validation checks were run, the students were 
sampled, and the Student Tracking and Session Attendance Forms were output from KeyQuest (exhibit 
2 and exhibit 3). Westat provided the lists of sampled students to schools two to four weeks before the 
scheduled testing date, depending on when the school provided the list of age-eligible students. A total 
of 6,677 students (an average of 40.5 per school) were randomly sampled. 
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Exhibit 1. Student listing form 




Program for International Student Assessment 2009 



P I S A 
USA 





School Name: 










Address: 










Telephone: 










Fai: 










List Prepared Bj: 










Total # of Students Listed: 









DIRECTIONS: Please list ALL students in your school in Grade 7 and higher born between 

Jul« 1. 1993 and June 30. 1994 



Student Name 


Birth 

Month 


Birth Year 


Gender 

f1=F;2=M1 


Grade 











































































































































































































































































































































































SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International Student Assessment (PISA), 2009. 
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Exhibit 2. Student tracking form 



School Name 


School ID 


School Coordinator 


Test Administrator 











Completed by PISA Home Office 
To be checked by SC 



Completed by PISA Home Office To be For use by SC or TA 

To be checked by SC completed if needed 



^ by SC 

t' ^ >> ✓ n t — ^ t r >, 



(i) 


(2) 


(3) 


(4) 


(5) (6) (7) (8a) (8b) (9) (10) (11) (12) 


Student Name 


Stratum 

ID 


School 

ID 


Student 

ID 


LST 

Line 

Number 


Grade 


Gender 

(F=1 ; M=2) 


Birth Date 

(MM-YYYY) 


Study 

Program 


SEN 


N/P 

PISA 


COMMENTS 
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Exhibit 3. Session attendance form 



School Name 


School ID 


School Coordinator 


Test Administrator 











STUDENT DETAILS 


SCHOOL DETAILS 


INSTRUMENT 
STUDENT DETAILS ALLOCATION 


ATTENDANCE* 


(1) 


(2) 


(3) 


(4) 


(5) (6) (7) (8) (9) (10) 


Student Name 


Stratum ID 


School ID 


Student ID 


Booklet StQ 

Grade Test StQ Comments 

Number Form 



Comments to add into column 10: 

o If a student was partially present (Code 2) during the 2 hours 
absent. 

o If a student was present or partially present for Test (Codes 1 

o If a student was present or partially present for Test (Codes 1 



of the test session, please record the length of time he or she was 

or 2) but was absent for Questionnaire part, please confirm this, 
or 2) but left the test booklet blank, please make a note. 



Codes to enter into column 8: 1 - present; 0 - absent; 

2 - partially present (absent 
for more than 10 min) 

Codes to enter into column 9: 1 - present; 0 - absent. 



Do NOT use code 2 in column 9. 
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As described in chapter 2, PISA 2009 international requirements stipulated that the school response rate 
target was 85 percent for all jurisdictions. A minimum of 65 percent of schools from the original sample 
of schools were required to participate for a jurisdiction’s data to be included in the international 
database. Jurisdictions were allowed to use replacement schools (selected during the sampling process) to 
increase the response rate once the 65 percent benchmark had been reached. 

PISA 2009 also required a minimum participation rate of 80 percent of sampled students from schools 
within each jurisdiction. A student was considered to be a participant if he or she participated in the first 
testing session or a follow-up or makeup testing session. Data from jurisdictions not meeting this 
requirement could be excluded from international reports. 

The PISA 2009 standards also required that nonresponse bias analyses need to be conducted if school 
response rates were less than 85 percent. NCES standards for assessment surveys stipulated that a 
nonresponse bias analysis is required at any stage of data collection with a weighted unit response rate 
less than 85 percent. The non-response bias analyses are provided in Appendix H. 

3.1 School Participation 

Table 2 provides the response status of original and substitute schools. 



Table 2. PISA 2009 schools by response status 





Original 


Substitute 


Total 




Number of 
schools 


Percentage 
of schools 


Number of 
schools 


Percentage of 
schools 


Number of 
schools 


Percentage of 
schools 


Total eligible schools 


236 


100.0 


74 


100.0 


310 


100.0 


Participating 


145 


61.4 


20 


27.0 


165 


53.2 


Refusal 


63 


26.7 


30 


40.5 


93 


30.0 


Ineligible/Closed 


28 


11.9 


3 


4.1 


31 


9.4 


Other (Pending, No contact) 


0 


0.0 


21 


28.4 


21 


6.8 



NOTE: Detail may not sum to totals because of rounding, 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International Student Assessment (PISA), 2009. 

Of the 165 participating schools, 145 schools were original schools and 20 schools were substitutes. At 
the close of recruitment, 21 substitute schools were in a pending status or had no contact. Contact with 
these schools had been stopped as the focus returned to approaching refusing original schools. 



The unweighted and weighted school response rates before and after replacements are shown in Table 3. 
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Table 3. School response rates 

Unweighted response rate percentage Weighted response rate percentage 

Before replacement 69,7 67,8 

After replacement 793 773 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International Student Assessment (P ISA), 2009. 

3.2 Student Participation 

Table 4 reports the participation status of students to be assessed including categories of non- 
participating students as defined by PISA. In total, 612 students (9.2 percent of students sampled) were 
coded as non-participating due to special education needs or having been withdrawn from school. 
Students excluded due to special education needs were considered non-participating as were students 
who were home- schooled or who had withdrawn. However, refusals were reported under students to be 
assessed because the calculation of the response rate includes in the denominator students who were 
absent and refused. 



Table 4. Status of sampled students 





Number of students 


Percentage of students 


Total students sampled 


6,677 


100.0 


To be assessed 


6,065 


90.8 


Non-participation 






Functional disability 


32 


0.5 


Cognitive disability 


213 


3.2 


Insufficient language ability 


85 


1.3 


Home schooled 


9 


0.1 


Withdrawn 


273 


4.1 



NOTE: Detail may notsum to totals because of rounding, 

SOURCE: U.S, Department of Education, National Center for Education Statistics, Program for International Student Assessment (P ISA), 2009. 

Table 5 reports the participation status of students to be assessed. Of the 6,677 students sampled, there 
were 6,065 students to be assessed. A total of 5,233 students were assessed, with 603 students absent 
and 229 students not assessed due to parent refusal. The overall unweighted student response rate was 
86 percent. The weighted student response rate was 87 percent. 



Table 5. Participation status of students to be assessed, PISA 2009 





Number of students 


Percentage of students 


Total students to be assessed 


6,065 


100.0 


Assessed 


5,233 


86.3 


Absent 


603 


9.9 


Parent Refusal 


229 


3.8 



NOTE: Detail may notsum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International Student Assessment (PISA), 2009. 

Four schools had participation rates below 50 percent. Eight other schools required make-up sessions to 
reach an acceptable response rate of 50 percent or better. 
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After experiencing difficulties achieving high levels of school participation in a number of international 
studies, NCES convened a task force prior to the beginning of the previous PISA administration (PISA 
2006) to make recommendations to improve participation rates in international studies. The plan for 
recruiting schools for PISA 2009 was based partially on the recommendations of the task force. The 
PISA 2009 school recruitment strategy included: (1) starting recmitment early, beginning at the start of 
the 2008 academic year, one year in advance of the data collection; (2) approaching schools directly, and 
sending information to relevant school districts and states; (3) assigning recruiters to specific schools, to 
strengthen the personal relationship between the recmiter and school staff; (4) providing cash incentives 
at both the school and student levels; and (5) holding a June 2009 Summer Conference in Washington, 
D.C. for school staff participating in PISA. 

4.1 Recruitment Materials 

The materials used for recmitment included a PISA study brochure, the Summary of Activities for 
Schools, a PISA Resource Kit on CD, letters to states, districts, schools, and parents, a fact sheet for 
parents, and a student invitation form. Examples of materials used at the state, district, and school level 
are provided in appendix B. Examples of materials used with parents and students are provided in 
appendix C. 

4.2 Recruitment of Schools 

Five recruiters with experience in gaining school cooperation were hired to recruit schools for the PISA 
Main Study. Each recmiter was responsible for approximately 47 schools. Recruiters were trained prior 
to beginning to recruit schools. 

During the last week of August 2008, packages were mailed to the state commissioners and testing 
directors, district superintendents, and district test directors. The packages contained: 

• A letter from the NCES Commissioner; 

• A PISA study brochure; 

• The Summary of Activities for Schools; 

• The Study Timeline; and 

• A guide to locating information about PISA (including sample assessment items and results). 

Twenty-six districts (12 percent of districts with original schools) required explicit approval before 
schools could be contacted. Formal research requests were prepared and sent to these districts. 

In addition, NCES called each state assessment director to inform them about PISA, and NAEP State 
Coordinators in each state with sampled schools were sent a folder of materials with a cover letter 
alerting them that PISA was beginning to contact schools. 

School packages were mailed to principals in mid-September with phone contact from recruiters 
beginning a few days after the mailing. The materials included a description of school and student 
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incentives. Participating schools received $200 and school coordinators received $100. Participating 
students received $20 and a certificate showing two and one-half hours of volunteer time. If the 
assessment was conducted out of school hours participating students received $35. 

Recruiters contacted schools by telephone to request their participation in PISA 2009. Recruiters also 
had frequent (at least weekly) individual and group contacts with the recruitment coordinator to discuss 
recruitment issues. 

4.3 Challenges in School Recruitment 

The PISA 2009 school recmitment was initially planned to last 4 to 5 months beginning in August 2008 
and ending by the end of 2008. By mid-October 2008 it was clear that gaining cooperation from schools 
was not progressing at a satisfactory rate. Of schools that did not agree to participate, many were 
unwilling to discuss the study with us or they refused to participate because of the amount of other, 
mandated testing and the loss of instructional time, or because of staff shortages. Reluctance from 
schools required the recruitment period to be extended well beyond what was planned. School contact 
began in mid-September 2008 and officially ended in the first week of November 2009 — one week 
before the end of the assessment window on November 13. 

Later in the spring of 2009, as recruitment staff were attempting to schedule the participating schools, 12 
schools that had initially agreed to participate early in the recruitment process (September and early 
October, 2008) withdrew their participation. The reasons provided tended to involve the reasons stated 
above as well as the local economic pressures facing schools. 

4.4 Solutions and Approaches Used with Refusing Schools 

Two initial approaches were implemented to increase participation: 

• PISA recruiters began making personal visits to schools. Staff visits continued through the end 
of the academic year. A total of 91 visits were made to 81 refusing or pending schools. 

• A letter was distributed to state chiefs through the Council of Chief State School Officers asking 
for state assistance. This resulted in several states contacting the project to either offer direct 
help or discuss the schools and approaches to use. 

Table 6 reports the results of visits and state assistance. Some schools received both a visit and state 
influence so these are not mutually exclusive results in all cases. Nor did joint contact always result in 
cooperation by the school. 



Table 6. Recruitment results of state contact and Westat personal visits 



Contact result 


Number of schools 


Percentage of schools 


Total state contacts 


29 


100.0 


State cooperation 


18 


62.0 


State refusal 


11 


38.0 


Total visits 


81 


100.0 


Visit cooperation 


28 


35.0 


Visit refusal 


53 


65.0 



NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International Student Assessment (PISA), 2009. 
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4.5 Student Recruitment 

Once the student sample was selected within a school, PISA staff worked with the school coordinator to 
obtain parental consent and school coordinators distributed student invitations to participate (provided 
in appendix C). Study recruiters and test administrators also worked with school coordinators to answer 
any student or parent questions, including sharing the PISA Fact Sheet for Parents (provided in 
Appendix C). 

There were three levels of parent consent: (1) explicit consent (parent consent agreement was required); 
(2) implicit consent (parents could opt out of study by returning a form); and (3) notification (parents 
were informed of the study). The level of consent used was determined by school or district 
requirements. 
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5.1 Test Instrument Design 

The 2009 assessment instruments were developed by international experts and PISA Consortium test 
developers, and items were reviewed by representatives of each jurisdiction for possible bias and 
relevance to PISA’s goals. The assessment included items submitted by participating jurisdictions as well 
as items that were developed by the Consortium’s test developers. 

The final assessment consisted of 102 reading items, 36 mathematics items, and 52 science items 
allocated to 13 test booklets. Each booklet was made up of 4 test clusters. Altogether there were 7 
reading clusters, 3 mathematics clusters, and 3 science clusters. The clusters were allocated in a rotated 
design to the 13 booklets. The average number of items per cluster was 15 items for reading, 12 items 
for mathematics, and 17 items for science. Each cluster was designed to average 30 minutes of test 
material. Each student took one booklet, with about 2 hours worth of testing material. Approximately 
half of the items were multiple-choice, 20 percent were closed or short response types (for which 
students wrote an answer that was simply either correct or incorrect), and about 30 percent were open 
constructed responses (for which students wrote answers that were graded by trained scorers using an 
international scoring guide). In PISA 2009, every test booklet included reading items. Some booklets also 
included mathematics items, science items, or both. 

5.2 Assessment Materials Development 

The materials for PISA 2009 in the United States included: (1) 13 different test booklets; (2) a School 
Questionnaire; (3) a Student Questionnaire; (4) a Test Administrator Manual; (5) a School Coordinator 
Manual; and (6) three separate coding guides for test items assessing reading literacy, mathematics 
literacy; and science literacy. The international English versions of these materials were developed by 
two consortia of organizations, one for the test booklets, coding guides, and manuals and the other for 
the student and school questionnaires. NCES was responsible for adapting the questionnaires, test 
booklets, and coding guides, and administration manuals for use in the United States. The goal of such 
adaptation was to ensure that the material used spellings and vocabulary that were most commonly used 
in the United States (but did not change meaning) and which reflected the actual U.S. administration 
plans. This involved: (1) changing spellings and vocabulary into common U.S. usage (e.g., changing 
“analyse” to “analyze” and “biscuits” to “cookies”); (2) adding a limited number of U.S. national items to 
the school and student questionnaires (e.g., adding items on racial/ ethnic group to the student 
questionnaire); and (3) adapting the administration manuals to reflect the U.S. plans for data collection. 

These adaptations were checked and reviewed by the international consortium through an iterative 
process that occurred over the period from March 25, 2009 through May 20, 2009. After the adaptations 
had been approved by the international consortium, the final versions of the 13 test booklets, the 
Student Questionnaire, and the School Questionnaire were produced and submitted to the two consortia 
for a “final optical scan.” 
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5.3 Printing of Instruments 

The PISA 2009 data collection instruments consisted of 1 3 test booklets, the Student Questionnaire, and 
the School Questionnaire. The Student Questionnaire was specially adapted to make it a scannable 
form. Copies of the School Questionnaire and Student Questionnaire are provided in appendix D. 

A detailed process for the review of test booklets and questionnaires was conducted by study staff and 
NCES to ensure a high quality of printed products. This process began with reviews of electronic copies 
of instruments and then reviews of hard copy versions. Numerous electronic and hard copy drafts were 
produced and reviewed by study staff and NCES. The final versions of all instruments were approved 
by NCES on July 20, 2009. 

5.4 Packaging and Distribution of Materials to Field Staff 

Final printed books and questionnaires were sent to a packaging facility, where security bar code labels 
were applied to the test books, and the booklets were bundled. 

To identify each test booklet, each test booklet was assigned a bar code ID number. The bar code ID 
numbers were printed on labels and then applied to the back cover of each test booklet. The bar codes 
were tracked for security purposes to ensure that all booklets sent out were returned. Each document 
type was also assigned an inventory number. 

The test booklets were then spiraled into bundles. Bundles of 13 booklets were created and spiraled in 
booklet 1 through 13 bar code sequence order. In addition, extra bundles of each of the 13 booklets 
were created for use by test administrators. The booklet type to be used was randomly assigned by the 
KeyQuest system and printed on the Session Attendance Form. Each bundle had a header sheet that 
indicated the bundle number and a range of the booklet IDs within the bundle. 

A customized packing list was created for each test administrator. Each test administrator was also 
assigned specific bundle numbers. This enabled study staff to identify where the test booklets should be 
at any time during the assessment. 

The distribution effort included packaging and mailing of test booklets, student questionnaires, and 
assessment related materials to test administrators. The test administrators were sent a supply of bulk 
materials in addition to a session box for each PISA assessment scheduled. 
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Data collection consisted of three major elements: 

• A School Questionnaire requiring approximately half an hour that was sent to schools prior to 
data collection and collected during the testing visit; 

• A student assessment administered in a two-hour testing session, with a short break in the 
middle; and 

• A Student Questionnaire taking approximately half an hour for students to complete. 

6.1 Pre-Assessment Contacts with School Staff 

Each participating school was required to designate a staff member to serve as school coordinator. 
School coordinators received a School Coordinator Handbook to use in preparing for the assessment. A 
significant portion of this document provided instruction on assembling a list of students and identifying 
students with special needs. 

The international version of the Handbook instructed schools to include a special needs code on the list 
of PISA eligible students. The United States adapted this by sampling the students first, and then asking 
that students with special needs be identified from the sampled students listed on the Student Tracking 
Form, combining this step with determining nonparticipation. This reduced the burden on the school 
by significantly reducing the number of students that needed to be evaluated. In many cases, school 
coordinators were required to consult other student records or meet with special education staff to 
identify these students’ specific needs and whether or not they could participate. 

Prior to the assessment, each school coordinator was contacted at least five times. 

1. Beginning in mid-August, 2008, school coordinators were sent an initial mailing containing the 
School Coordinator Handbook, a hard copy student listing form, and a cover letter outlining the 
process for assembling and submitting a list of students to the study. 

2. This mailing was followed by an e-mail outlining the process in more detail and describing 
options for the submission of the list of students. An electronic student listing form was 
attached in this email. 

3. After the student list was received, study staff processed the list following PISA guidelines and 
using the required international sampling software, KeyQuest. This process resulted in the 
production of a Student Tracking Form indicating which students in the school had been chosen 
to participate. A second mailing was then sent to the school coordinator containing the Student 
Tracking Form, the School Questionnaire, the school coordinator payment, student invitations, 
and a cover letter. The cover letter explained the next steps of identifying students with any 
special education needs (SEN), indicating any students who would not be able to participate 
(either due to an SEN, parent refusal, or the student transferring out of the school), solidifying 
arrangements for the assessment, and discussing parent consent and the importance of student 
participation. 

4. A follow-up e-mail was sent the day after the mailing had been received. This e-mail reviewed 
the SEN and participation information and consent materials. The consent letters requested by 
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the school were attached to this e-mail with the direction that schools could use these items as a 
template and change as necessary to meet the needs or requirements of the school. 

5. The final contact prior to the assessment was the pre-assessment call conducted by the test 
administrator approximately one week before the assessment. Using a Pre-Assessment Call 
Checklist, the test administrator reviewed with the school coordinator the logistics for 
assessment day, the Student Tracking Form, and anticipated student response. 

As a general rule, test administrators were instructed to make a “courtesy call” to the school coordinator 
one to two days before the assessment. The courtesy call was implemented to determine if student 
participation was a problem and if the test administrator could assist in any way and to cover any last- 
minute questions or concerns with the school coordinator. 

In many cases, additional contacts were made in fielding questions from school coordinators via the toll- 
free phone line or the project e-mail. These contacts generally dealt with questions or clarifications 
about student sampling. Almost all of the student lists required some level of verification or further 
contact with the school. 

6.2 Data Collection Training 

Training for data collection was held September 11—12, 2009, in Rockville, Maryland. Twenty-four test 
administrators attended the training. 

One week before training, each test administrator received a Test Administrator Manual containing the 
instructions for pre-assessment work and conducting the assessment in schools. Each test administrator 
was given four hours of home-study time to familiarize themselves with the PISA procedures. The test 
administrators also completed a short quiz prior to training. This was designed to take the test 
administrator through the manual and become familiar with specific information about PISA 
procedures. 

The training was administered simultaneously in separate rooms to two small groups of 12 test 
administrators. The format of the training was constructed around detailed PowerPoint presentations 
with interactive discussions to include all participants, as well as hands-on exercises and activities to 
engage participants and practice procedures. The training agenda is provided in Appendix C. 

Day 1 focused on an introduction and overview of the study, the materials required, the tracking forms, 
their completion, pre-assessment call with the school coordinator, and booklet and questionnaire 
labeling. Day 2 focused on the assessment day activities: arriving at the school and setting up the 
assessment area, administering the assessment, packing the session boxes for return, and distributing 
student incentives. Throughout the training the importance of maintaining security of materials and the 
confidentiality of respondents was emphasized. 

Only test administrators attended the 2-day training. Test administrators were responsible for training 
their assistant administrators prior to the start of the first assessment in their areas. Each test 
administrator was supplied with training scripts and necessary exercises. Prior to the assistant 
administrator training, each assistant administrator was supplied with an Assistant Administrator Manual, 
similar to the Test Administrators Manual, but focusing primarily on the assessment day activities. The 
test administrators were instructed to spend approximately one-half day training the assistant 
administrators. 
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6.3 Data Collection Approach 

The study employed 30 test administrators, one assigned to each work area. Test administrators were 
assigned to one of two field managers who coordinated and monitored their work. During the testing 
period, test administrators reported to their field manager almost on a daily basis. To assist test 
administrators, 30 assistant administrators were hired to create two-person assessment teams. These 
assistants assisted in labeling forms, setting up the testing areas, and monitoring students during the 
testing sessions. The assistant was only responsible for test administration if two session areas were 
required. 

Test administrators were responsible for: 

a) familiarizing themselves with the Test Administrator Manual; 

b) successfully completing training prior to the start of assessments; 

c) hiring and training assistant administrators; 

d) conducting pre-assessment calls with school coordinator within 2 weeks of the assessment; 

e) conducting follow-up contacts with school coordinator 1-2 days before the assessments to 
ascertain if any problems with student attendance; 

f) ensuring that each student received the correct testing materials; 

g) administering the test in accordance with the internationally-specified instructions, including 
following the administration script; 

h) ensuring the correct timing of the testing sessions; 

i) completing the Session Administration Form; 

j) recording student participation on the Session Attendance Form; 

k) ensuring that the test booklets, student questionnaires, Student Tracking Form (STF), Session 
Attendance Form and Session Report Form, and completed school questionnaire, were returned 
to Pearson typically within 24 hours; 

l) reporting any issues or problems with the assessment to the field manager immediately after the 
assessment; and 

m) updating the Field Management System (FMS) with final student counts and changes from the 
STF, notes on the assessment, tracking numbers of the session boxes and disposition code for 
the school. 

The assessment administration consisted of three segments. The students were assessed in two 
segments, each 1 hour in length. These were to be administered on the same day, with a short break of 
approximately 5 minutes in between. After the second hour the students received another break and 
were administered the student questionnaire. This questionnaire took approximately 30 minutes. 

The timing of the assessment sessions was as follows: 

a) approximately 1 0 minutes for preparation, including preparation of students, reading 
instructions, distribution of test booklets, etc.; 

b) 1 hour for answering Part 1 of the test booklet; 

c) a short break of approximately 5 minutes; 

d) 1 hour for answering Part 2 of the test booklet; 

e) a short break of no more than 1 0 minutes; 

f) approximately 30 minutes for the student questionnaire; and 

g) 10 minutes for collecting materials, distributing incentives and ending the session. 

Students were told to bring something to read in case they finished early. 
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6.4 Data Collection Activities 

The PISA 2009 data collection was administered between September 23 and November 19, 2009. Table 
7 shows the number of assessments that were completed in each month. Ten of the assessments were 
conducted outside of regular school hours (after school or on Saturday). 



Table 7. School assessments by month 



Month 


Number of schools 


Percentage of schools 


Total school assessments 


165 


100.0 


September 


37 


22.4 


October 


91 


55.2 


November 


37 


22.4 



NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International Student Assessment (PISA), 2009. 

Make-up sessions were scheduled only if the schools had a very low student response rate or if the 
logistics of holding a make-up were favorable and the response rate was below 85 percent. Seventeen 
make-up sessions were conducted. 



2009 Data Files and Database with U.S. Specific Variables 



21 



Chapter 7. 
Data Management 



7.1 Receipt Control 

PISA materials were returned by the test administrators to data management staff from September 22nd 
through November 23th. When a shipment arrived, the materials were forwarded to a 
receiving/ opening area. The school ID, receipt date, and the return shipper’s tracking number were 
recorded in a receipt system. Data management staff notified test administrators if a school’s materials 
were not received within three days after its test date. 

Using specifications provided by the managers, staff were trained to check each school’s materials using 
the Student Attendance Form (SAF) returned with the booklets. They verified: 

• All student booklets and student questionnaires for students listed on the SAF were returned to 
Pearson; 

• Information coded on the booklet covers matched the SAF, i.e., school number, student ID, and 
participation status; 

• A Student Tracking Form (STF) and Session Report Form (SRF) had been returned with each 
school’s materials; and 

• The school questionnaire was returned. 

If one of these three forms (STF, SAF, SRF), student booklets or student questionnaires were missing, 
data management managers were notified. Data management managers then contacted the data 
collection managers and test administrators for resolution. 

Once a school’s material was verified as complete, booklets and questionnaires were sorted into separate 
work units. Unused or unassigned booklets were separated and the security bar codes were recorded 
into a file. After processing was completed, this file was combined with the bar codes of the completed 
booklets. Staff compared the bar codes to those shipped out to identify if any books were not returned. 
All booklets and questionnaires were returned by data collectors. 

7.2 Batching 

After check-in was completed, the completed booklets were forwarded to the batching area. Each batch 
was assigned a unique batch number. This number, created on a Work Flow Management (WFM) 
system for all documents, facilitated the internal tracking of the batches and allowed departmental 
resource planning. The batch number was written on a control sheet and was present on all batch/ stack 
headers. Each batch type had its own unique control sheet that listed all of the workstations 
(departments) the group of work needed to pass through before reaching a clean post-edit. A clean 
post-edit meant that everything had been scanned, key entered, and edited correctly based on the 
specifications written by software development staff and the International Codebooks. 

ACER guidelines required that a specialized batching system be used for test booklets. This system 
generated “blue batches” (those booklets to undergo multiple coding) and “yellow batches” (those to 
undergo regular coding). 
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The WFM indicated when the batch had moved to the next process — data entry and/ or scanning. Data 
management staff was able to view what process the batch was in at any given time using this system. 
The student booklet and School Questionnaire batches/ carts were forwarded to the Data Input area for 
manual key entry. The Student Questionnaire batches/ carts were sent to the Scanning area. 

7.3 Data Entry 

Depending on the PISA document, one of two methods was used to transcribe PISA student data to an 
electronic form. 

• The test booklets and school questionnaire data were key entered into a data file using the 
Falcon data entry system, which is a mainframe data system. 

• The data on the student questionnaires was entered using 99091 optical-scanning equipment. 

Data entry staff entered 6,792 test booklets (including unused booklets) and 163 School Questionnaires 
into the Falcon system (two School Questionnaires arrived late and were entered manually during file 
creation). There were 6,792 Student Questionnaires scanned using the 99091 scanning equipment. 

For the test booklets and School Questionnaire, Software Development staff created key-entry 
specifications based on the national booklets and the international codebooks. Data Input senior staff 
created key entry screens on the Falcon System using the specifications and the actual books as guides. 

Ten qualified, permanent trained staff keyed the student booklets and School Questionnaires. A 
standard practice to ensure the data was captured correctly required the student booklets and School 
Questionnaires to be keyed twice (keyed and then verified) by two different Data Input staff members. 

If the information from the keyer did not match the verifier, the field came up in error. The discrepancy 
was resolved by a senior team member. 

7.4 Scanning 

Scan development staff created a scan program for the Student Questionnaire using specifications 
provided to them by the Software Development staff. The specifications were based on the 
International Codebook and the layout of the actual questionnaires. The scanned data values captured 
were coded in the same manner as the student booklets and school questionnaires. 

To properly scan the booklets, the spine (folded or left side of the booklet) needed to be removed. A 
slitter machine was used for this process. 

A scan operator entered the WFM batch number and application name from the control sheet into the 
machine to “call-up” the correct scan program. This insured that the program being run accepted the 
correct, and only the correct, book/ form types for that batch of work. Diagnostic sheets were run 
through the scanner first to ensure the scanner was picking up the gridded circles correctly. Quality 
control sheets were also randomly inserted into each stack of books by the scan operator. This was 
another quality check to help ensure the documents were processed accurately within stringent control 
limits. 

The images of constructed-response response items in the student questionnaire were saved as Tagged 
Image Files (TIF). The area of the page that needed to be clipped was defined prior to scanning through 
the document definition process. The fields from unreadable pages were coded “X” as a flag for 
resolution staff to correct. These images were sent to the Image editing area for manual key entry. Any 
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image document or sheet unreadable by the image scanning system was taken to a flatbed scanner to be 
scanned into the system. 

Once scanning of all documents in the batch was complete, the scan operator closed out the batch and 
the batch of work was then forwarded to the next workstation listed on the control sheet (Image 
Editing). 

7.5 Data Editing 

To ensure all data entry and editing functions had been set up properly and per the International 
Codebooks, the first batch of the student assessment booklets, Student Questionnaires, and School 
Questionnaires went through a quality control check. Software Quality Specialists checked to make sure 
the data was collected and edited according to the specifications given to the programmers. They 
verified that the output file matched the responses recorded in the booklets and questionnaires. 

Assessment Booklets and School Questionnaires 

The data from the assessment booklets and questionnaires was run through a Pre-Edit Program and 
edited according to specifications supplied by the Software Development staff based on the PISA 
International Codebook. If there were problems, the paper or image editing staff corrected them to 
ensure the data were clean. 

The Editing staff reviewed a computer-generated edit log and the area of the source document that was 
noted as being suspect or as containing possible errors in the assessment booklets and school 
questionnaires. The Editing staff checked this piece of information against the PISA test booklet or 
school questionnaire. 

The corrected edit log was forwarded to the Key Entry staff for processing (re-keying). When all 
corrections were entered and verified for a batch, an extract program pulled the corrected records into a 
mainframe dataset. At this point, the mainframe edit program was initiated. If there were further errors, 
a new post-edit listing was printed and the cycle was repeated. These edited batch files were uploaded to 
the mainframe. 

Student Questionnaires 

After a student questionnaire batch had been scanned, it was routed to the Image Editing area. Staff 
reviewed errors identified in each batch through an on-line editing system. Image clips requiring edits 
were routed to on-line editing stations. 

For rapid resolution, the edit criteria for each item in question appeared on the editing screen along with 
the suspect item. Corrections were made immediately. The system employs an edit/verify system which 
ultimately means that two different people view the same suspect data and operate on the item 
separately. The verifier made sure the two responses were the same before the system accepts that item 
as being correct. If the editor cannot determine the appropriate response, he or she escalated the suspect 
situation to a supervisor. 

After the data from the assessment booklets and questionnaires had been entered, edited, and no more 
errors existed, the batch was considered clean. Project staff received a “processed documents” report 
from those batches on a daily basis or on-demand. This information taken from a Process Control 
System (PCS) which listed the number of documents processed for each school received. 
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7.6 Occupational Coding of Student Questionnaires 

Students were asked to report their parents’ or guardians’ occupations on the Student Questionnaire. 
These responses were coded during the processing of the student questionnaires. Student responses in 
the student questionnaire for occupational coding were captured during the scanning process. Once a 
batch was designated as clean, a report listed the student ID and the occupational responses. Pearson 
staff coded each response using the International Standard Classification of Occupations (ISCO) manual 
(International Labour Organization (ILO) version 1988) occupational codes located in the Data 
Management Manual. Two reports were printed for each batch. Two staff members coded them 
separately. These were entered separately into a mainframe file. A comparison was run and if any 
disagreements in codes were identified, a third person decided on the correct code. Development staff 
corrected the file and continued to mn the comparison until all discrepancies were resolved. 

Study staff coded 13,584 responses due to double-coding of each response. 

7.7 Coding of Test Booklet Constructed-Response Items 

There were 53 reading items, 15 mathematics items, and 19 science items that required students to 
construct a response. The constructed-response assessment items for PISA 2009 were coded during 
November and December 2009. The study used two (2) Scoring Directors, four (4) supervisors, and 24 
coders to code the responses from U.S. students. Both Scoring Directors attended coding training in 
Brussels, Belgium. 

Coding Materials 

Separate coding guides were prepared for reading, mathematics, and science constructed-response items. 
International coding guides were provided by ACER, and these guides were adapted to reflect U.S. 
adaptations of the test booklets. 

The Scoring Directors received photocopied examples of U.S. student responses in early October, and 
used these responses to supplement the international coding guides. Unlike the neatly typed responses 
on a single page in the international coding guides, these practice materials looked like actual student 
responses in booklets (for example, handwritten, written under the prompt, other prompts on the page). 
These additional practice materials were reviewed and approved by NCES before scorer training began. 

Scorer Training 

Supervisor training occurred on November 12 and 13, 2009 in Mesa, Arizona. The Scoring Directors 
trained the four scoring supervisors on the items that each trained and supervised during the coding 
window. 

Training and scoring began on November 16, 2009. A brief introduction to the PISA project was given 
to all scorers as a large group. All scorers were experienced scoring personnel. An overview of the PISA 
assessment included general coding guidelines, inter-rater reliability and rate expectations, scorer bias and 
grade-level considerations regarding the PISA items. These experienced scorers already understood 
confidentiality and quality issues and the importance of coding accuracy. All scoring staff signed the 
PISA 2009 confidentiality agreement prior to coding. 

The scorers were then separated into three teams and the training began with the first cluster for each 
team. Reading had one team of 16 scorers. Mathematics and science each had one team of four scorers. 
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PISA training and coding occurred on a cluster basis. Thus, the team trained on all items within the 
cluster and then coded all responses for those items before the trainer introduced the next cluster. Since 
all of the PISA scorers were experienced, the study had requested a deviation from ACER in September 
2009, in order to code all of the responses within the cluster for one student before moving onto the 
next student’s responses. This request was approved. 

Scorers read the item and any accompanying materials, such as a reading passage and ancillary materials 
(e.g., cartoons, graphs, maps, brochures). The supervisor then read the item aloud, and then read the 
description for each code level category from the coding guide. Supervisors discussed the coding guide 
and any student examples within the coding guide. Scorers were encouraged to ask questions, but the 
supervisor discouraged over-generalizations until scorers had seen more examples. 

After reviewing the coding guide and the examples, scorers independently coded the practice examples. 
Practice examples generally consisted of: (a) the range of codes on the scoring guide; (b) different levels 
of responses within a code level (i.e., high “2” and low “2”); (c) a mix of solid and borderline examples; 
and (d) examples of common student responses. Practice examples were ordered randomly. Some of 
the practice examples were from the international coding guides while others were from U.S. students. 

Upon completion of the practice examples, the supervisors reviewed the practice sets with their teams. 
The supervisors asked scorers to explain why examples were scored as they were, and compared the 
codes given by individual scorers with the pre-determined codes. 

During training, supervisors monitored each scorer on the individual items within each PISA cluster, 
noting any who had difficulty. Supervisors kept notes on all coding decisions and changes made to 
practice sets during training of the items. Coding guides and training materials with annotations were 
complete and correct and reflected actual training for each item. Information from flip charts and/ or 
wall charts developed by supervisors and used to document coding decisions were included in the 
documentation. The supervisors made certain that all scorers were taking notes on such decisions 
throughout the training sessions. 

Paper Flow 

Coding of test booklets occurred in an internationally-defined sequence. The yellow batches (those not 
requiring multiple scoring) were coded first. Then the second, third, and fourth clusters in the blue 
batch booklets were coded. Finally, the first cluster in the blue batch booklets (those requiring coding by 
four scorers) were coded. The coding of multiple- coded clusters followed the prescribed procedure by 
ACER, in which the same set of coders scored every booklet in a batch. The single-coded clusters were 
coded by any scorer on the team. 

Score sheets were created for each cluster in a booklet. The first cluster in each blue batch booklet had 
four score sheets provided for first, second, third and fourth coding. The remaining clusters had only 
one score sheet per cluster. 

Scoring Directors scheduled the scoring of clusters to avoid crossover between subjects wherever 
possible (as all three subjects occurred in the same booklets, all three teams needed access to a booklet at 
some point in the process). Warehouse staff routed batches to appropriate scorers as directed by 
Scoring Directors. 
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Scoring Supervision 

Careful, ongoing monitoring of scorers was crucial to all scoring. While this monitoring involved 
Scoring Directors, the “front line” was the supervisor for each scoring team. The supervisors ensured 
that backreading took place at the specified rate and that it was careful and thoughtful. 

Backreading allowed Scoring Directors and supervisors to read student responses that had previously 
been scored by each team member. The Supervisors reviewed the code assigned and evaluated whether 
or not it was accurate. 

The supervisors were required to backread (check the scoring of) all scorers, with a target of backreading 
five percent of all scorer work done on a daily basis. However, this level of backreading could be 
adjusted depending upon a variety of factors (e.g., rate of scoring, item type). Backreading began almost 
immediately after scoring started and continued throughout the scoring process. This allowed the 
supervisors to identify coding problems quickly. When beginning backreading a new item, the 
supervisors considered what he or she observed during the training and practice scoring sessions. The 
supervisors paid particular attention to the accuracy of scorers who had difficulty applying the coding 
guide consistendy during training and would backread those scorers first. 

Additional training on an individual or group basis was sometimes needed based on information gleaned 
from backreading. Some scorers needed to adjust their interpretation of the coding guide or specific 
types of student responses. Supervisors shared examples of student responses as part of this follow-up 
training. 

Training and coding of the mathematics items was completed on December 4, 2009, science items on 
December 7, 2009, and reading items on December 9, 2009. Table 8 shows the number of hours 
required to complete the coding process. 



Table 8. Hours required for booklet coding 





Number of staff 


Average hours worked 


Total days worked 


Total number of hours 
worked 


Supervisors 


4 


8 


18 


576 


Coders 


24 


8 


16 


3,072 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International Student Assessment (P ISA), 2009. 

Score Recording 

A paper-based scoring system, ePS, was used to record these scores of constructed-response items. 
Software Development staff created individual score sheets for each cluster within each of the 1 3 
booklets. They had only one cluster’s items printed per score sheet, with the bar code identifier of the 
booklet preprinted. 

Once a score sheet had been coded by a scorer, the score sheet was scanned using the ePS system. Study 
staff scanned 6,794 score sheets using the ePS scanning system. During scanning, the scoring system 
notified the technician of any missing codes, blank responses, or out-of-range codes on the score sheets. 
This system also produced two reports: sheets-left-to-scan and inter-rater reliability. The “Sheets left to 
Scan” report insured all score sheets had been coded and scanned. 

After coding was complete, the responses captured were uploaded to a mainframe file. Software 
Development staff then merged these with the multiple-choice responses. The combined file was then 
imported into KeyQuest. 
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Inter-Rater Reliability (IRR) 

The study used the one hundred responses in the blue batches of each booklet that were coded by four 
different scorers to calculate the IRR percentage. About one third of all booklets were selected to be 
placed in the blue batches. Only the items in the first cluster within these booklets were read by four 
scorers. All of the yellow batch booklets (as well as the clusters in the second, third, and fourth positions 
in the blue batches) were coded before the IRR are calculated. 

The inter-rater reliability for specific items ranged from 92.2 to 100.0 percent, with 63 percent of the 
items having reliabilities of 90.0 percent or above. The reliabilities of specific items are shown in 
Appendix E. 

7.8 File Creation and Consistency Checks 

All data files were loaded into the KeyQuest software, the international data management software, as 
directed in the Data Management Manual. Before the upload process, Software Quality Specialists 
verified the files were in the correct format based on the KeyQuest International Codebooks. Staff 
members ran several validation reports to check for mismatches and duplicate records. 

7.9 Data Confidentiality Safeguards 

The Education Sciences Reform Act of 2002 explicitly requires that NCES protect the confidentiality of 
all those responding to NCES-sponsored surveys so that no individual respondent can be identified. 
More specifically, NCES Standard 4-2, Maintaining Confidentiality (NCES 2002), provides guidelines for 
limiting the risk of data disclosure for data released by NCES. Data disclosure occurs when an individual 
respondent has been identified through the use of the survey item responses and other external data 
sources. The following discussion describes the procedures used to reduce the risk of data disclosure for 
PISA 2009, in accordance with the guidelines specified in NCES Standard 4-2. 

All students and schools participating in the PISA do so with the assurance that their identities will not 
be disclosed. Confidentiality procedures in place included the following: (1) all employees with access to 
the data signed affidavits of data confidentiality; (2) questionnaires were sealed by students after 
completion; and, (3) names of students and schools were removed by field staff from the assessment 
booklets, the questionnaires and all other related materials, and replaced with unique identification 
numbers. In addition to data collected direcdy from schools and students, additional information was 
used during the PISA sampling, data collection, and weighting processes and these variables too were 
considered as part of the review to determine disclosure risk levels. 

In addition, after the School Questionnaire and Student Questionnaire files were successfully entered 
into Keyquest, study staff conducted disclosure analyses to determine if individual schools or students 
could be identified using data from other publicly available data sources. While no public data 
collections identify students by name, three publicly available data collections do identify schools by 
name. These are: the Common Core of Data (CCD), a detailed public school listing; the Private School 
Survey (PSS), a detailed private school listing; and, the QED data collections produced by Quality 
Education Data, Inc., a privately owned education research firm. The QED data contain a school-based 
file that provides demographic information for both public and private schools along with the names of 
the schools. Thus, there is some possibility that schools at least, and perhaps students as well, could be 
identified if comparisons of these data sets with the PISA data set allowed the identification of schools. 
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It might be possible to identify PISA schools by taking variables from the PISA school data and 
searching the publicly available data files for schools with a matching profile. However, since the 
variables in the PISA data files were obtained from responses to the school questionnaire, for the most 
part, exact profile matches are unlikely. Even then, one would not know for certain whether any of the 
matched schools were the actual PISA schools, or whether the match had simply arisen by chance. 

Nevertheless, school matching analyses were undertaken using probabilistic matching algorithms 
approved by the IES Disclosure Review Board (DRB) for use in disclosure analyses. These algorithms 
identify schools with some potential for identification. In order to provide further protection, elements 
of the data from schools identified as “disclosure risks” in this way were perturbed using the procedures 
approved by the DRB. After perturbation, the data were subjected to another round of analyses to 
ensure that the potential for identification no longer existed. 

An additional measure was taken to reduce further the risk of disclosure of an individual respondent. 
This measure is referred to as “data swapping”, a DRB requirement that reduces risk by modifying 
microdata. In data swapping, a probability sample of records is paired with other records on the file 
using selected characteristics, and then some identifying variables are swapped between pairs of records. 
The sampling rate for PISA swapping was designed to protect the confidentiality of the data without 
affecting the usability of the dataset. All questionnaire data (school and student) were involved in the 
swapping. This method is an effective way of keeping as much valuable data as possible while protecting 
respondent identity. Swapping preserves the univariate frequencies, means, and variances, although it 
may affect multivariate relationships a little. Pre- and post-swapping percentage distributions 
(unweighted and weighted) and correlations were reviewed to ensure data quality was maintained. 

7.10 Data Editing and File Delivery 

After the data files were successfully entered into Keyquest and disclosure review activities had been 
completed, study staff implemented the data editing procedures required by ACER. On the basis of 
those procedures, staff identified a few minor data inconsistencies (e.g., missing forms for students 
marked as present on the Student Attendance Form). Based on review of the data, these inconsistencies 
were resolved and the dataset was forwarded to ACER on January 27, 2010. 

After receipt, ACER implemented some additional data checking procedures and asked for clarification 
on a few selected items. Study staff provided responses, and ACER made some final editing changes to 
the dataset. 

For the International Coding Review (a reliability check of coding across countries), the international 
consortium sent a list of 80 student IDs each from Booklets 8 and 12 which were multiple-coded in the 
United States and which would be recoded by the international contractor. The study provided an 
electronic copy of the first cluster in each of these booklets to ACER via their secure website on 
February 10, 2010. 
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8.1 International Data File Cleaning and Editing 

ACER cleaned each of the national data files to ensure that data cleaning was standardized among all 
participating countries. ACER’S role at this point was to check that the international data structure was 
followed, check the identification system within and between files, correct single case problems 
manually, and apply standard cleaning procedures to questionnaire files. Results of the data cleaning 
process were documented and shared with the national project managers and included specific questions 
when required. The national project manager then provided ACER with revisions to coding or solutions 
to anomalies. ACER then compiled background univariate statistics and preliminary classical and Rasch 
item analysis. 

8.2 Missing Data 

PISA does not impute missing information for questionnaire variables. The international database and 
the U.S. database contain four kinds of missing data codes that are used across all countries. “Missing” 
data occur when a respondent is expected to answer an item but no response is given. Responses that 
are “invalid” occur in multiple choice items for which an invalid response is given. An item is coded 
“not applicable” when it is not possible for the respondent to answer the question (e.g., an assessment 
item not included in the student’s booklet or an item to be skipped based on a previous item). Finally, 
test booklet items that are “not reached” are consecutive missing values starting from the end of each 
test session. All four kinds of missing data are coded differently in the PISA 2009 database. 

8.3 Weights for U.S. Data 

The use of sampling weights is necessary for the computation of statistically sound, nationally 
representative estimates. Survey weights adjust for the probabilities of selection for individual schools 
and students, for school or student nonresponse, or for errors in estimating the size of the school or the 
number of 15-year-olds in the school at the time of sampling. Survey weighting for all jurisdictions 
participating in PISA 2009 was carried out by Westat, as part of the PISA consortium. 

The internationally defined weighting specifications for PISA 2009 included base weights and 
adjustments for nonresponse. The school base weight was defined as the reciprocal of the school’s 
probability of selection. (For substitute schools, the school base weight was set equal to the original 
school it replaced.) The student base weight was given as the reciprocal of the probability of selection 
for each selected student from within a school. 

These base weights were then adjusted for school and student nonresponse. The school nonresponse 
adjustment was done individually for each jurisdiction using implicit and explicit strata defined as part of 
the sample design. In the case of the United States, two variables were used for stratification: school 
control and Census region. The student nonresponse adjustment was done based on students’ explicit 
stratum; within cells, grade and gender were also used to define nonresponse adjustment. Trimming 
factors at the school and student levels were used to reduce the size of large weights, since large weights 
can substantially increase sampling variance. All PISA analyses were conducted using these adjusted 
sampling weights. 
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8.4 Scaling of Student Test Data 

Thirteen versions of the PISA test booklet were used, each containing a different subset of items. The 
fact that each student completed only a subset of items means that classical test scores, such as the 
percent correct, are not accurate measures of student performance. Instead, scaling techniques were used 
to establish a common scale for all students. For PISA 2009, item response theory (IRT) was used by the 
international contractor to estimate scores for reading, mathematics, and science literacy, as well as for 
three reading literacy subscales: accessing and retrieving information, integrating and interpreting, and 
reflecting and evaluating. 

IRT identifies patterns of response and uses statistical models to predict the probability of answering an 
item correctly as a function of the students’ proficiency in answering other questions. With this method, 
the performance of a sample of students in a subject area or sub-area can be summarized on a simple 
scale or series of scales, even when students are administered different items. 



1 The combined reading literacy scale is made up of all items in the three subscales. However, the combined reading 
scale and the three subscales are each computed separately through Item Response Theory (IRT) models. Therefore, the 
combined reading scale score is not the average of the three subscale scores. 
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9.1 PISA 2009 International Datasets 

Data from PISA 2009 for all countries can be obtained from the OECD. At the time of this report’s 
publication (July 2011), these data were available from http:/ /pisa2009. acer.edu.au/. Users can either 
select to download entire files, choose only selected variables, or run simple queries. Files available for 
downloading include the following (note that the parent questionnaire, the information communication 
technology questionnaire, and education career questionnaire were not administered in the U.S.): 

Questionnaires 

Student questionnaire 
School questionnaire 
Parent questionnaire 

Information Communication Technology (TCT) questionnaire 
Education Career (EC) questionnaire 
Reading for School (RFS) questionnaire 

Codebooks 

Codebook for student questionnaire data file 
Codebook for school questionnaire data file 
Codebook for parent questionnaire data file 
Codebook for cognitive item response data file 
Codebook for scored cognitive item response data file 

SAS control files 

SAS syntax to read in student questionnaire data file 

SAS syntax to read in school questionnaire data file 

SAS syntax to read in parent questionnaire data file 

SAS syntax to read in cognitive item response data file 

SAS syntax to read in scored cognitive item response data file 

SPSS control files 

SPSS syntax to read in student questionnaire data file 

SPSS syntax to read in school questionnaire data file 

SPSS syntax to read in parent questionnaire data file 

SPSS syntax to read in cognitive item response data file 

SPSS syntax to read in scored cognitive item response data file 

Data sets in TXT format (compressed) 

Note that some of these files are very large. 

Student questionnaire data file 
School questionnaire data file 
Parent questionnaire data file 
Cognitive item response data file 
Scored cognitive item response data file 
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Compendia 

The compendia provide the distribution of students according to the variables collected through the student, information 

communication technology, parent and school questionnaires. The performance means per category are also provided. 

Compendium for the student questionnaire 

Compendium for the school questionnaire 

Compendium for the parent questionnaire 

Compendium for the ICT and EC questionnaire 

Compendium for the RFS questionnaire 

Compendium for the cognitive item responses 

9.2 U.S. National Data Files 

Data collected in the United States for PISA 2009 can be downloaded from the international site or from 
the NCES website. The files on the international website contain data for all countries, including the 
United States. The NCES files, which include only data for the United States, are as follows: 

Student data 

• The data are contained in US_ST09.TXT. This file contains questionnaire items and derived 
variables and index scores based on the student questionnaire; plausible values for overall 
reading, reading subscales, science scale, and the mathematics scale from the assessment; and 
student sampling weights and replicate weights. There are 5,233 cases in this file. Since the data 
are hierarchical (students are clustered with schools), each student record contains identification 
variables that enable the user to merge the school data with the student data, using the variable 
SCHOOLID. 

• An SPSS syntax file, US_ST09.SPS 

• A SAS syntax file, US_ST09.SAS 

• A codebook file (US PISA 2009 Student Codebook.PDF) that includes variable names, variable 
location and format information, variable labels, question text, values, and frequencies. 

School data 

• The data are contained in US_SC09.TXT. This file contains items from the school questionnaire, 
derived variables and index scores based on the school questionnaire, and the school sampling 
weight. There are 1 65 cases in this file. 

• An SPSS syntax file, US_SC09.SPS 

• A SAS syntax file, US_SC09.SAS 

• A codebook file (US PISA 2009 School Codebook.PDF) that includes variable names, variable 
location and format information, variable labels, question text, values, and frequencies. 

Scored cognitive (assessment) item data 

• The data are contained in US_ASSESM09.TXT. This file contains scores of student responses to 
each item in the assessment. There are 5,233 cases in this file. It should be noted that data are in 
two response formats: (1) valid scores = 0, 1; 0 = no credit, 1 = full credit; and (2) valid scores 
= 0, 1 , 2; 0 = no credit, 1 = partial credit, 2 = full credit. Also note that unlike the international 
files, there is no cognitive item response file (unscored responses). The majority of the items 
have not been released, so there is little descriptive information about them. The variable label 
for S131Q04D in the file is S131Q04D in the file is “SCIE - P2006 (broken link) Good 
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Vibrations (Q04).” This means that because of changes in responses to the item, responses on 
this item should not be compared to responses on the same item in previous PISA 
administrations. 

• An SPSS syntax file, US_ASSESM09_S.SPS 

• A SAS syntax file, US_ASSESM09_S.SAS 

• A codebook file (US PISA 2009 Scored Cognitive Assessment Codebook.PDF) that includes 
variable names, variable location and format information, variable labels, question text, values, 
and frequencies. 

U.S. questionnaires 

• The U.S. version of the student questionnaire is in the file: PISA_MS09_StudentQ_USA.PDF 

• The U.S. version of the school questionnaire is in the file: PISA_MS09_SchoolQ_USA.PDF 

PISA 2009 Technical Report and User’s Guide 

This document, PISA Technical Report and User's Guide.PDF, contains information on the conduct of 
PISA in the United States. 

9.3 National and International Variables 

The U.S. national data contain both the “international variables” (questionnaire and assessment variables 
used by all countries) and a few “national variables” (questionnaire variables used only in the United 
States). Note that the same assessment items were used by all countries. There are also some variables 
that appear in the international files that are missing for U.S. cases. These include three questionnaires 
not used in the United States: the parent questionnaire, the Information Communication Technology 
(TCT) questionnaire, and the Education Career (EC) questionnaire. Variables used only in the United 
States and those not used in the United States are shown in tables 9 and 10. 
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Table 9. Variables used only in the United States 

Variable name Questionnaire item wording 

Student Questionnaire 

ST02A01 
ST05N01 
ST06 
ST06A01 
ST06A02 
ST06A03 
ST06A04 
ST06A05 
ST07A01 
ST08A01 
ST10A01 
ST11A01 
ST20Q15 
ST20Q16 
ST20Q17 
ST30A08 

School Questionnaire 

SC05A01 Number of public schools in your area that compete for your students 

SC05A02 Number of private schools in your area that compete for your students 

SC10A01 Approximately, how many computers are available forall students in yourschool for educational purposes? 

SC25A01 Regarding yourschool, which of the following bodies exert a direct influence on decision making about 

staffing, budgeting, instructional content and assessment practices? (Local education agency or local 
school board) 

SC27A01 Approximately what percentage of students at this school last year were eligible for free- or reduced-price 

lunch through the National School Lunch Program? 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International Student Assessment (P ISA), 2009. 

Table 10. Variables not used in the United States 

Variable name Questionnaire item wording 

Student Questionnaire 

ST02Q01 Which one of the following programmes are you in? 

School Questionnaire 

SC01Q13 Do you have the following grade levels in yourschool? Grade 13 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International Student Assessment (PISA), 2009. 

International variables recoded from U.S. questionnaire variables 



How long have you been in this school? 

Which best describes you? (I am Hispanic or Latino./! am not Hispanic or Latino.) 
Which of these categories best describes your race? 

White 

Black or African American 
Asian 

American Indian or Alaska Native 
Native Hawaiian or Other Pacific Islander 
Did you attend pre-school? 

Did you attend kindergarten? 

Have you ever repeated a grade? (Kindergarten) 

What is the highest grade or level of schooling you expect to complete? 

Which of the following are in your home? A guestroom 

Which of the following are in your home? A high-speed Internet connection 

Which of the following are in your home? A musical instrument 

How often are you involved in the following reading activities? (Text-messaging) 



Three international questionnaire items on the Student Questionnaire needed to be rewritten to be 
applicable for U.S. questionnaires, and thus required international recoding. 



• The international question ST05Q01 (Did you attend <ISCED level 0>?) was adapted into two 
U.S. questions ST05N01 and ST05N02 (Did you attend pre-school? and Did you attend 
kindergarten?). The U.S. values were recoded in ST05Q01 to reflect international values. 

• The international questions ST10Q01 and ST14Q01 (What is the highest level of education 
completed by your mother/ father?) included five options, one of which (ISCED level 3B, 3C) 
was not relevant in the U.S. Thus the U.S versions of these variables (ST14N01 and ST18N01) 
have four valid responses rather than five and have different value labels than the international 
versions. 
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9.4 Variable Names 

The variable names created by the SPSS and SAS syntax files are those used on international data sets. It 
should be noted that on the Student Questionnaire, those variable names do not correspond with the 
item numbers on the questionnaire because a number of U.S.-only items were inserted near the front of 
the questionnaire. For convenience, the item numbers on the U.S. Student Questionnaire and School 
Questionnaire are listed in parentheses at the end of the variable name (e.g., “Mother’s occupation 
(Q13)”). 

9.5 Derived Variables 

The international contractors for PISA have developed a number of derived variables for use in their 
analyses, and these variables have been included on the student and school files. They appear after the 
questionnaire variables and have variable names that do not contain numerals. The explanation for 
many of these variables is included in Appendix E, which is abstracted from two volumes of the 
international report (OECD 2010c and OECD 2010d). A more complete explanation of these variables 
will be provided in the international PISA 2009 Technical Keport (forthcoming). 
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10.1 Special Considerations — Plausible Values and Replicate Weights 

Three aspects of the design of PISA need careful attention in any analysis. The first stems from the 
sample design. Schools and students had unequal but known probabilities of selection. As a 
consequence, to generalize to the population sampled, analyses will need to apply the sampling weights 
provided in the file. 

The second aspect to be considered also stems from the sampling design and bears on the calculation of 
standard errors. Since the sample design is complex (a two-stage, stratified cluster design), most software 
packages, operating on the assumption of a simple random sample, will produce biased estimates of 
standard errors. Special procedures that use the replicate weights contained in the data file are called for, 
and are described in detail in the PISA Data Analysis Manual, SPSS, Second Edition (OECD 2009a). These 
procedures are implemented in several stand-alone software packages (WesVar, AM, and SUDAAN, for 
example), but can also be implemented in SPSS using the macro posted on the NCES website, or in SAS 
using the information provided in the PISA Data Analysis Manual, SAS , Second Edition (OECD 2009b) . 
Standard errors produced in published reports were estimated using Fay’s method of Balanced Repeated 
Replicates (BRR) with 80 replicates and the Fay coefficient set to 0.5. That method should be specified 
when using SUDAAN or other stand-alone software packages to analyze the PISA data. 

The third aspect arises from the design of PISA and the use of plausible values in analysis. In PISA, as in 
many national or international assessments, students are not administered every assessment item. Each 
item then has missing student responses, though these are missing by design. As a consequence, it is not 
possible to estimate scores for individual students. Instead, the results of individual students are 
aggregated to produce scores for groups of students (e.g., all U.S. students, U.S. female students, etc.). 
For analysis purposes, PISA datasets include sets of five “plausible values” for each student for each 
overall subject area score and each subscale score. The plausible values are intended to represent the 
estimated distribution of scores of students similar to the given student in terms of responses to the 
assessment items and background questionnaire items. 2 What this means for analyses is that, in effect, 
any analyses involving the achievement scores must be done five times, once for each plausible value, 
and then the results must be averaged. A special provision also needs to be made in the estimation of the 
standard errors and is best done using the SPSS or SAS macro developed for this purpose. Again, these 
issues are discussed in the PISA Data Analysis Manual, SPSS, Second Edition (OECD 2009). 

10.2 Nonresponse Bias 

Detailed analyses were conducted to determine if nonresponse at either the school or questionnaire item 
level resulted in apparent biases in the results. The results indicated that school nonresponse to the 
study resulted in limited apparent bias of results. On the other hand, analysis of nonresponse to specific 
questionnaire items indicated that for three items on the School Questionnaire (Q9ab, Q9bb, and Q9cb) 
and ten items on the Student Questionnaire (Q12d, Q12e, Q12f, Q15a, Q15c, Q15d, Q19a, Q19b, Q19c, 



2 For theoretical and empirical justification of the procedures used, see Mislevy (1988). For more information about the 
methodology used in PISA see OECD (forthcoming). 



2009 Data Files and Database with U.S. Specific Variables 



37 



Chapter 10. Using the PISA 2009 Data Files 



and Q19d) there was significant nonresponse (more than 15 percent) and statistically significant 
differences between respondents and nonrespondents. (The full nonresponse bias analysis report is 
included in Appendix H. 

10.3 Merging School and Student Data 

The PISA sample was designed to yield a representative sample of 1 5-year-old students enrolled in 
schools; the school sample was designed to optimize the selection of these students. In these 
circumstances, it is usually recommended that the school data should be disaggregated across students 
and school attributes be treated as “student characteristics” for the purposes of the analyses. This 
disaggregation can be accomplished by merging the school-level data to the student file using 
SCHOOLID and the resulting file analyzed at the student level using the replicate weights (W_FSTR1— 
W_FSTR80). 
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Comparison of the PISA 2009 and NAEP 2009 Reading Assessments 

Prepared by the International Activities Program, 

National Center for Education Statistics 

December 2010 



A.1 Background 

In the United States, nationally representative data on student achievement come primarily from two 
sources: the National Assessment of Educational Progress (NAEP) — also known as “The Nation’s 
Report Card” — and U.S. participation in international assessments, including the Program for 
International Student Assessment (PISA), the Trends in International Mathematics and Science Study 
(TIMSS), and the Progress in International Reading Literacy Study (PIRLS). While the international 
assessments may appear to have significant similarities with NAEP, each was designed to serve a specific 
purpose and each is based on a separate and unique framework and set of assessment items. Thus, each 
gives a somewhat different view of U.S. student performance. 

In December 2010, the National Center for Education Statistics (NCES) is releasing results from the 

2009 administration of PISA, an assessment of 15-year-olds’ reading, mathematics, and science literacy 
(Fleischman et al. 2010). In PISA 2009, reading literacy was the major domain; therefore, detailed 
information about U.S. student performance in reading will be available. The PISA release follows the 
March 2010 release of the NAEP 2009 4*- and 8*-grade reading assessment results and the November 

2010 release of the NAEP 2009 12*-grade reading assessment results (NCES 2009; NCES 2010). In 
anticipation of questions about the similarities and differences between the PISA and NAEP reading 
assessments, and what each can tell us about U.S. students’ reading skills, NCES prepared this paper, 
which discusses aspects of each assessment. In particular, the paper discusses the purposes, target 
populations and reporting levels, and content assessed by each. For NAEP, the paper focuses on the 
reading assessments for students in grades 8 and 12, as these are the two target populations closest to 
PISA’s target population of 15-year-olds. To examine and compare the content measured by PISA and 
NAEP, NCES convened a panel of reading experts 1 to compare how PISA and NAEP define reading, 
aspects of the texts used as the basis of the assessments, and the reading processes required in each 
assessment. This information is intended to help the press and others understand the similarities and 
differences between the assessments and to help identify what PISA and NAEP each contribute to the 
overall knowledge base on U.S. student reading performance. 

A.2 Purposes of PISA and NAEP 

The goals of the two assessments have subtle but important distinctions with regard to U.S. curricula. 

International assessments, such as PISA, support comparisons of student performance among countries 
but provide less information for within-U.S. comparisons. NAEP supports comparisons of student 
performance among states, public and private schools, student demographic groups, and a set of urban 



1 The experts included Patricia Alexander (University of Maryland), Gina Biancarosa (Stanford University and University 
of Oregon), Michael Kamil (Stanford University), Pamela Mason (Harvard University), and Junko Yokota (National 
Louis University). The meeting was facilitated by staff from the NCES International Activities Program in a joint effort 
with staff from the NCES Assessment Division. 
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public school districts. 2 Both PISA and NAEP are conducted regularly to allow the monitoring of 
student outcomes over time. PISA is conducted every three years and NAEP is conducted, for some 
subjects (including reading), every two years; for some subjects, every four years; and for other subjects, 
less frequently. 3 

PISA provides internationally comparative information in the United States on the reading, mathematics, 
and science literacy of students at an age that, for most countries, is near the end of compulsory 
schooling. The objective of PISA is to measure the “yield” of education systems, or what skills and 
competencies students have acquired and can apply in these subjects to real-world contexts as they near 
the transition from compulsory schooling. PISA’s literary concept, which applies to the reading, 
mathematics, and science assessments, emphasizes the mastery of processes, understanding of concepts, 
and application of knowledge and functioning in various situations. By focusing on literacy, PISA 
assesses what students have learned in and outside of the school environments. 

NAEP reports information on achievement in reading and other subjects at the 4*-, 8*-, and 12*-grade 
levels across the country. NAEP assessments are based on assessment frameworks and achievement 
levels (i.e., Basic, Proficient, and Advanced) established by the National Assessment Governing Board. 
NAEP assessment frameworks and achievement levels are based on the collaborative input of a wide 
range of experts and participants from the government, education, business, and public sectors in the 
United States. The frameworks are intended to be a blueprint for the assessment (specifying what should 
be assessed), and the achievement levels act as performance standards for each subject area and grade, 
showing what students should know and be able to do (NCES 2009, p. 4—5). 

The focus of NAEP on subject matter expectations in the United States distinguishes it from PISA, the 
content of which is determined in collaboration with other countries. The focus in PISA on the yield of 
education systems and the application of competencies in real-world contexts distinguishes it from 
NAEP, which measures school-based performance and abilities to read and understand written texts and 
to interpret and use what students have read in ways that are appropriate to the type of text and 
situation. 

A.3 Target Populations Assessed by PISA and NAEP 

The students assessed represent different target populations. 

PISA and NAEP are both sample-based assessments, meaning that each assessment administered is to a 
sample of students (rather than to all students) and the results are generalized to the larger population. 
However, each assessment defines the population to which it is generalizing, and thus from which the 
sample is drawn, differently. One distinction between main NAEP and PISA is that NAEP uses grade- 
based samples, whereas PISA uses an age-based sample. These choices relate to the purpose of each 



2 In “main NAEP,” students in grades 4, 8, and 12 are assessed; in “long-term trend NAEP,” students ages 9, 13, and 17 
are assessed. This paper focuses on main NAEP (on which the 2009 NAEP results are based); thus, all statements about 
NAEP in this paper refer to main NAEP. 

3 PISA is on a 3-year cycle, with one domain being featured as the major domain every 9 years. In 2000 and 2009, 
reading literacy was the major domain in PISA. Main NAEP currently assesses 4th- and 8th-grade reading and 
mathematics every 2 years. Every 4 years, NAEP assesses 12th-grade reading and mathematics, as well as 4th-, 8th, and 
12th grade science. Other subjects, such as writing, civics, economics, and U.S. history, are also assessed, but less 
frequently. In 2009, there were 18 public school districts that participated in the Trial Urban District Assessment 
(TUDA) reading, mathematics, and science assessments. 
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program — NAEP, to report on student achievement based on what students learn by a specific grade in 
school; and PISA, to describe the yield of education systems toward the end of compulsory schooling. 

The PISA target population is all 15-year-old students. In 2009, this included all students who were 15 
years and 3 months to 1 6 years and 2 months at the beginning of the testing period (fall 2009) and who 
were enrolled in school, regardless of grade level or full- or part-time status. The majority of respondents 
in the U.S. PISA 2009 sample were in 10* grade (68.5 percent), but some were in the 11* (20.3 percent), 
9* (10.9 percent), or another grade (0.3 percent). The NAEP target populations are all students in the 4*, 
8*, and 12* grades, and NAEP reflects the performance of U.S. students enrolled in these grades. Thus, 
the PISA results are for students who are mosdy in grades between those being tested for NAEP (the 8* 
and 12* grades), and closer in grade proximity to those taking the NAEP 8*-grade assessment because 
of the timing of the respective assessments (with PISA given earlier in the school year than NAEP). 

A.4 Reporting Levels and Sample Sizes for PISA and NAEP 

PISA and NAEP are designed to provide results at different levels of aggregation and, as a result, have different levels of 
precision. 

PISA and NAEP are both designed to provide information about U.S. students’ performance aggregated 
to the national level and for subgroups of the population (e.g., subgroups defined by gender and 
race/ ethnicity). NAEP, however, is also designed to provide reading results for individual states and 
some large urban districts. 4 Under the federal Elementary and Secondary Education Act of 2002, states 
receiving Title I funds are required to participate in NAEP reading, as well as mathematics, at the 4*- 
and 8*-grade levels. No requirements are placed on states to participate in PISA or other international 
assessments; and while states or districts could opt to participate in PISA and receive state- or district- 
level results, to date none have. 

The reporting requirements for NAEP and PISA have implications for sample sizes and, in turn, for 
which subgroups’ results can be reported and for the precision of the estimates. The NAEP national 
sample comprises the state and district samples and thus is extremely large. For example, the sample size 
for the grade 8 NAEP 2009 reading assessment was more than 160,000 students. In contrast, in 2009, 
PISA assessed 5,233 students. Because of its large sample size, NAEP is able to reliably measure 
achievement for more subgroups than PISA can. For example, while both NAEP and PISA can report 
achievement for students based on their racial and ethnic classifications, in 2009 the PISA sample did 
not include a sufficiently large sample size to report achievement for students who are identified as 
American Indian/Alaska Native or Native Hawaiian/Other Pacific Islander. Moreover, because of 
NAEP’s large sample size, it can detect smaller differences between subgroups or over time than can 
PISA. For example, while the standard error associated with the NAEP 2009 reading national mean 
score for grade 8 was 0.3 (NCES 2009), the standard error for the PISA 2009 reading mean score was 
3.7 for the United States (Fleischman et al. 2010). 

Related to sampling is the degree of inclusion of students with special or language needs. Both NAEP 
and PISA strive to be inclusive and ultimately achieve similar inclusion rates, although their specific 
policies differ. PISA is designed to be as inclusive as possible and requires that no more than 5 percent 
of the target population be excluded from testing. Exclusions are allowed at both the school level (e.g., a 
geographically remote school) and within schools at the student level, including students with a 
functional disability, intellectual disability, or insufficient language experience (defined as non-native, 



4 State- and district-level results are also provided for some other subjects. 
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limited proficient speakers with less than 1 year of instruction in the testing language). Currently, there 
are no special accommodations provided for students taking PISA. NAEP’s policy endeavors to assess 
all students selected as part of its sampling process and allows a range of accommodations, as necessary, 
for students with disabilities (SD) or English language learners (ELL). Accommodations include 
modifications in presentation format, response formats, test-taking setting, timing, or other aspects, as 
well as direct and indirect linguistic support. The weighted exclusion rate was 5 percent of students in 
PISA in 2009. The exclusion rate in the NAEP 8*-grade assessment in 2009 was 4 percent of students. 

A.5 What Is Measured by the PISA and NAEP Reading Assessments? 

PISA and NAEP measure some similar aspects of reading, but there are differences in how reading is defined in the 
frameworks, in the types of passages presented to students, and in the types of cognitive skills required of students. 

To examine the similarities and differences in the content assessed by PISA and NAEP, NCES 
commissioned a panel of experts to examine and compare the PISA and NAEP reading frameworks, 
passages, and items. The panel looked at how each assessment defined reading; how the domain was 
organized in the frameworks; the nature, length, and difficulty of the reading passages; the format of the 
items used; and the cognitive processes in which students were asked to engage. 

A.5.1 Definitions of Reading 

There is overlap between the NAEP 2009 and PISA 2009 definitions of reading (see exhibit 1). The 
definitions for both assessments identify reading as a constructive process that involves interaction 
between the reader and the text, and both focus on understanding and using written text. There are 
subde differences, however. PISA’s definition emphasizes the use of reading for personally defined goals 
and growth and for participation in society, while the NAEP definition reflects the notion that readers 
draw on the ideas and information they have acquired from text to meet a particular purpose or 
situational need. The NAEP reading framework calls for the 12* grade assessment to address the 
preparedness of 12*-graders for postsecondary education and training, although NAEP does not 
currendy report results on a preparedness scale. 

Exhibit A.l Definitions of reading 

Reading literacy in PISA 2009 

Reading literacy is understanding, using, reflecting on, and engaging with written texts in order to achieve 
one’s goals, to develop one’s knowledge and potential, and to participate in society 
(OECD 2009, p. 23). 

Reading in NAEP 2009 

Reading is an active and complex process that involves understanding written text; developing and 
interpreting meaning; and using meaning as appropriate to type of text, purpose, and situation (National 
Assessment Governing Board 2008, p. 2). 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), PISA 2009 Assessment 
Framework: Key Competencies in Reading, Mathematics and Science, 2009; and U.S. Department of Education, National Centerfor Education Statistics, National 
Assessment of Educational Progress(NAEP), Reading Framework forthe 2009 National Assessment of Educational Progress, 2008. 
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A.5.2 Framework Organization and Assessment Features 

There are similarities in how PISA and NAEP organize the reading domain, although both PISA and 
NAEP have unique features and requirements not included in the other assessment. 

As shown in exhibit 2, the 2009 NAEP reading framework is based on a two-dimensional matrix with 
text types as one dimension and cognitive processes (“cognitive targets”) as the other. These can be 
thought of as “what students read” and “what students do with what they read.” PISA’s framework also 
organizes the assessment around texts and cognitive processes (“aspects”). PISA’s text dimension, 
however, includes four text taxonomies — type, format, medium, and environment. Type is largely 
analogous to NAEP’s “text type.” Format distinguishes between “continuous” texts — that is, text that is 
formed by sentences organized into paragraphs — and noncontinuous texts — that is, texts that are 
composed of less than sentences (e.g., lists, tables, graphs, diagrams, advertisements, schedules, 
catalogues, indexes and forms) (OECD 2009). Medium and environment are used because the PISA 
2009 reading assessment included an assessment of student reading literacy with electronic texts; medium 
distinguishes between print and electronic text 55 , and environment distinguishes between authored and 
message-based text. 6 NAEP does not currently include electronic texts in its reading framework or 
assessment. 

PISA also uses a unique third dimension, the reading situation, which distinguishes the range of contexts 
or purpose for which reading takes place. NAEP, on the other hand, has its own unique feature: an 
assessment of “meaning vocabulary,” which refers to students’ ability to apply meaning to words vital for 
comprehending the overall passage. The framework specifies that each NAEP passage will have 
approximately two items that focus on meaning vocabulary. 

A.6 Reading Passages 

The reading passages selected for inclusion in PISA and NAEP represent the individual framework and 
design of each assessment. Both assessments strive to cover a wide range of text types, difficulty, and 
topics. 

Both assessments distinguish a range of text types, which are somewhat, but not perfecdy, aligned. For 
example, NAEP’s literary category includes fiction, poetry, and literary nonfiction and is more expansive 
than PISA’s corresponding category, narration. Also, there is no clear counterpart to PISA’s 
“description” category, which includes documents that typically provide an answer to “what?” questions, 
such as a depiction of a place or a schedule. NAEP texts in the informational category include 
exposition, argument and persuasion, and procedural texts and documents (e.g., news articles, research 
reports, historical documents, persuasive essays, and position papers). Across grade levels, NAEP 
incorporates increasingly complex text structures and features, genre/type of text, and author’s craft. 
Although NAEP includes some noncontinuous material, it is only used as augmentation, embedded in 
continuous material, at the 8*-grade level, and there are only a few stand-alone examples in the 12*- 
grade assessment. PISA, on the other hand, makes heavier use of noncontinuous material, including, in 
particular, texts that fall into PISA’s exposition, argumentation, and transaction categories. Even the 



5 The PISA 2009 electronic reading assessment was an international option; the United States did not participate. 

6 The distinction between authored and ?nessage-based text is whether the reader has the potential to influence the content 
of the site. An authored environment does not allow the reader to modify the content (e.g., home pages, government 
information sites, news sites, etc.) whereas a message-based environment allows the reader the opportunity to add to or 
change the content (e.g., e-mail, blogs, chat rooms, etc.) (OECD 2009, p.29). 
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continuous texts in those categories are often drawn from activities that a 15-year old student might 
engage in during a daily routine and may come from a wide range of sources that are not strictly 
academically grounded (e.g., from a mainstream newspaper versus from a student’s educational 
magazine). 

PISA is designed to cover a wide breadth of what students read and the purpose for reading, which is 
not always in school but outside of school as well. PISA includes both continuous and noncontinuous 
text, as well as the range of types described in table 1 . The most common text type in PISA is exposition, 
which includes almost one-third of the passages. The least common text type in PISA is narration, which 
is represented in about 10 percent of the passages. 

The framework for NAEP also addresses the different kinds of reading materials students will encounter 
both in and outside of the classroom and describes NAEP as an “assessment of varied reading skills.” 
The broad categories of literary and informational texts are identified in two ways: first, by the different 
purposes for which literary and informational texts are read, and, second, by structural differences 
between literary and informational texts that mark the text and help readers understand what they are 
reading. Passages in the 8th-grade assessment are evenly distributed between these two main categories. 
At the 12th-grade level, about one-quarter of the passages are literary and about three-quarters are 
informational. This is intended to mirror the distribution of the kinds of texts students encounter as they 
progress through the education system. 

The PISA reading assessment includes 29 passages, and the NAEP 8*- and 12*-grade assessments 
include 16 and 17 passages, respectively. In PISA and NAEP, each student receives only a subset of the 
passages in each assessment. In NAEP, both the 8*- and 12*-grade assessments include some passages 
that are used at another grade as well — e.g., there is a subset of 8th-grade passages that is also used in the 
4*-grade assessment and a subset that is used at 12* grade. Also, NAEP pairs some passages. Students 
are presented two related passages and are asked inter-textual questions, as well as questions specific to 
each passage. PISA does not have paired passages in the same sense, although some PISA passages 
include multiple parts and may even include different text types or formats, and inter-textual questions 
may be included. In PISA, students respond to 2 to 5 items per passage, whereas in NAEP students 
respond to between 9 and 1 1 items per passage or pair of passages. 
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Exhibit A.2 PISA and NAEP reading framework dimensions and features 



Dimensions 


PISA 


NAEP 


Texts 


Type 


Text tvpes 




Narration 


Literary: 




Exposition 


fiction 




Argumentation 


literary nonfiction 




Instruction 


poetry 




Transaction 


Informational: 




Description 

Format 

Continuous, noncontinuous 
Medium 
Print, electronic 
Environment 
Authored, message-based 


exposition 

argumentation/ persuasive text 
procedural texts and documents 


Cognitive processes 


Aspects 


Copnitive targets 




Access and retrieve 


Locate and recall 




Integrate and interpret 


Integrate and interpret 




Reflect and evaluate 


Critique and evaluate 


Situation 


Contexts 

Personal, Public, Occupational, 
Educational 


Category does not exist 


Other features 




Meaning vocabulary 



SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), PISA 2009 Assessment 
Framework: Key Competencies in Reading, Mathematics and Science, 2009; and U.S. Department of Education, National Center for Education Statistics, 
National Assessment of Educational Progress(NAEP), Reading Framework for the 2009 National Assessment of Educational Progress, 2008. 



Related to the issue of text types, there is another important but subtle distinction between NAEP and 
PISA. The NAEP framework explicitly emphasizes the authenticity of text and notes a commitment to 
selecting high-quality, authentic stimulus materials that students are likely to encounter both in school 
and out of school. To this end, NAEP sets minimum passage lengths for inclusion and makes very few 
edits to the original texts. Although PISA is intended to measure authentic tasks, the PISA framework 
does not explicitly emphasize the use of existing, intact text. It is constrained in some ways by its 
international nature, as passages must be applicable across a wide range of cultures and languages. 
Therefore, while passages are selected to represent a range of texts and applicability in real-world 
settings, more manipulation and editing of passages is used than is in NAEP. Also, the 2009 NAEP 
reading framework explicitly required that the selection of passages be informed by readability analyses, 
such as the ones described in the next section; although readability analyses had played a role in passage 
selection prior to 2009, the 2009 framework explicitly called for their use. 

Length 

PISA passages are notably shorter than NAEP 8*- and 12*-grade passages, averaging 354 words to 
NAEP’s 924 and 1,174 words per passage or pair of passages, 7 respectively (see table 1). Passages or 
pairs of passages in NAEP range from 219 words to 1,429 words in the 8* grade and 771 to 1,429 words 



7 The reading passage length calculations reported here for NAEP include combined scores of paired passages and 
individual reading passage scores. 
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in the 12* grade, compared with a range of 53 to 758 words in PISA. Thus, even the longest passage in 
PISA is shorter than the average passage length for each of the two NAEP grades. The NAEP 
framework specifies passage length by grade level to represent what students encounter in their in-school 
and out-of-school reading, to ensure usage of strategic reading skills, to ensure that approximately 10 
distinct items can be generated from the passage, and to ensure that the structural patterns of the 
passages are supportive of the range of text types and that the items cover the range of cognitive 
processes (National Assessment Governing Board 2008, p. 28). PISA does not have a similar 
requirement for passage length as part of its framework. While PISA’s noncontinuous texts tend to be 
shorter than its continuous texts (see appendix table A), the presence of noncontinuous texts alone does 
not account for these differences. Rather, it is likely that these differences are driven by the differing 
framework requirements, constraints, and purposes of the assessments described earlier. 



Table A.l. NAEP 2009 and PISA 2009 reading passage word and item analyses 



Length of passage and items per 


NAEP grade 8 1 


PISA 


NAEP grade 12 2 


passage 


(N = 13) 


(N =29) 


(N = 13) 


Averaqe number of words 


923.6 


354.4 


1173.5 


Ranqe of words in passaqes 


219-1,429 


53 - 758 


771 - 1,429 


Averaqe number of items 


10.0 


3.6 


10.1 


Ranqe of number of items 


9-11 


2-5 


9-11 



'This column presents averages and ranges for all reading passages categorized as grades 4/8, 8, and 8/12. 

2 This column presents averages and ranges for all reading passages categorized as grades 8/12 and 12. 

NOTE: In NAEP a student might be presented with a single passage ora pair of passages; length analyses were calculated based on the passage or pair of 
passages a student received. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress(NAEP) 2009 Reading 
Assessment; and Organization forEconomic Cooperation and Development, Program for International Student Assessment (PISA) 2009 Reading Literacy 
Assessment 

Passage Difficulty 



Several readability measures (i.e., the Dale-Chall Formula, Flesch Reading Ease, Flesch-Kincaid Grade 
Level, and FORCAST Formula) were used to compare passage difficulty between PISA and NAEP (at 
the 8* and 12* grades) (see table 2). These analyses excluded passages of less than 250 words (which 
included 3 poetry passages in NAEP and 8 passages in PISA) because of the lower reliability of applying 
the formulas to passages of such length. 8 In general, PISA passages were somewhat closer in difficulty to 
12*-grade NAEP than to 8*-grade NAEP. The PISA passages tended to cover a broader range of 
readability or grade levels than did the NAEP passages. 



Using the formulas best suited to continuous text (Dale-Chall and Flesch-Kincaid) — although applied to 
both continuous and noncontinuous passages — passages from the NAEP 8*-grade assessment 
corresponded, on average, to a 7*-grade level according to Dale-Chall and 8*-grade level according to 
Flesch-Kinkaid, with a range that extended from 5* grade to 12* grade. The average grade level for 
NAEP 12*-grade passages was the 7* grade according to Dale-Chall and the 9* grade according to 
Flesch-Kincaid. The range of grade levels extended from 5* grade to 1 3* grade. The average grade level 
for PISA passages was the 8* grade for Dale-Chall and the 9* grade for Flesch-Kincaid. The range of 
grades extended from 3rd grade to 15* grade. Using the formula best suited to noncontinuous texts 
(FORCAST) — although again applied to both passage types — the NAEP 8*-grade and 12*-grade and 
PISA passages all averaged at the 10*-grade level. 



8 Eighth-grade NAEP reading passages included all individual reading passages categorized as grades 4/8, 8, and 8/12. 
Twelfth-grade NAEP reading passages included all individual reading passages categorized as grades 8/12 and 12. All 
readability analyses were calculated based on individual reading passages and do not take into account cases in which 
passages were paired. 
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When the passages for each assessment are separated by format (continuous and noncontinuous), other 
differences emerge (see appendix table B). Within assessments, PISA’s continuous passages tend to be 
more difficult than its noncontinuous passages. On the other hand, NAEP’s relatively few strictly 
noncontinuous passages at the 12*-grade level are more difficult than its continuous texts at that grade 
level. In fact, NAEP 12*-grade noncontinuous passages are, on average, more difficult than PISA 
noncontinuous passages, which are more numerous, and the NAEP 12*-grade noncontinuous passages 
are among the most challenging in either assessment. 
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Table A.2. NAEP 2009 and PISA 2009 reading passage readability analyses 



Readability measure 


NAEP grade 8 1 
(N = 14 ) 


PISA 
(N = 21) 


NAEP grade 12 2 
(N = 16) 


Description of measure 


Dale-C hall Formula 


The Dale-C hall Formula uses a familiar words list common to 
students and rates the text against it as well as the sample's total 
number of words and sentences. The more familiar words there are 
in a text, the easier the text is scored. 


Averaqe Dale-C hall qrade 


6.9 


7.6 


7.4 


Range of Dale-C hall grades 


5, 2-8, 3 


5. 8-9. 9 


5. 4-9. 4 


Flesch Reading Ease 


The Flesch Reading Ease measure is based on the number of 
words, syllables, and sentences in adult reading materials. Reading 
Ease scores fall between 0 and 100, with a lower number indicating 
more difficult material, The readability scores correspond to the 
following readability levels: very difficult (0-29), difficult (30-49), 
fairly difficult (50-59), standard (60-69), fairly easy (70-79), easy 
(80-89), and very easy (90-100). 


Average readability score 


69.4 


60.7 


62.4 


Range of readability scores 


51-85 


32-88 


41-83 


Flesch readability identifier 


Standard 


Standard 


Standard 


Flesch-Kincaid Grade Level 


The Flesch-Kincaid Grade Level formula is most reliable when 
used to assess upper elementary and secondary materials. Similar 
to the Flesch Reading Ease measure, it is based on the number of 
words, syllables, and sentences in a text, but with slightly different 
weighting. 


Average grade level 


7.5 


8.7 


8.8 


Range of grade levels 


4.8-11.5 


3.2-15.1 


5.3-13.4 


FORCAST Formula 


The FORCAST Formula focuses on the number of single-syllable 
words present in a text. It is usually used in evaluating 
questionnaires, forms, tests, and job materials notin narrative form 
and often absent of any end punctuation, such as periods or 
question marks, 


Average FORCAST grade 


9.5 


10.2 


9.8 


Range of FORCAST grades 


7.9-10.6 


8.2-11.7 


8.3-11.7 



iThis column presents averages and ranges of all individual reading passages categorized as grades 4/8, 8, and 8/12. 

2 This column presents averages and ranges of all individual reading passages categorized as grades 8/12 and 12. 

NOTE: Only passages with word counts over 250 were included in the readability analyses. Excluded passages included three from NAEP (which include the poetry reading passages) and eight from PISA. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2009 Reading Assessment: and Organization for Economic Cooperation and Development, 
Program for International Student Assessment (PISA) 2009 Reading Literacy Assessment 
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How Interchangeable Are the PISA and NAEP Passages? 

The expert panel was asked to determine how well NAEP and PISA passages would fit in the 
other’s framework or, in other words, the likelihood that the passages of one assessment could 
appear in the other. The expert panel reviewed a sample of about 70 percent of the passages from 
each assessment selected to represent the full range of the frameworks in terms of text type and 
length. 10 They found that PISA passages tended to fit better to the NAEP framework than NAEP 
passages did to the PISA framework, although a substantial number of passages from both 
assessments were deemed not interchangeable (see table 3). About half of the NAEP 8*-grade and 
two-thirds of the NAEP 12th-grade passages that were reviewed did not fit within the PISA 
framework. Just over two-fifths of the PISA passages that were reviewed did not fit within the 
NAEP framework at either the 8* or 12* grade. The most typical reason for lack of fit of NAEP 
passages was the prominence of “author’s craft” in NAEP. (“Author’s craft” refers to the specific 
techniques used by an author to relay an intended message.) NAEP texts such as poetry or 
rhetorical narratives would be difficult to translate into the various languages required by PISA, as 
would maintaining the tone and quality of a text as the author intended. The experts concluded that 
some PISA passages would not appear in the NAEP assessment because there was too much 
disconnected text (or presentation of multiple stimuli not strictly related), the texts were not 
authentic enough, or the passages were simply too short. 



Table A.3. Percentage distribution of NAEP 2009 and PISA 2009 passages that fit the other 
assessments framework 



Fits/ Does not fit 


PISA to NAEP 
(Grade 8 or 12) 


NAEP to PISA 


Grade 8 


Grade 12 


Fits other framework 


57 


50 


33 


Does not fit other framework 


43 


50 


67 



NOTE: The judgment of the expert panel, rather than any specific formula, was used assess Tit" and to calculate the percentage. Analyses are based on 
a representative sample of roughly 70 percentof the passages from each of the three assessments. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Database of expert panel ratings, J une 2010. 

A.6.1 Item Format 

Both NAEP and PISA include multiple-choice items from which students choose one correct 
answer. All NAEP multiple-choice items include four response options, whereas PISA multiple- 
choice items include four or five response options (see table 4). In addition to traditional multiple- 
choice items, PISA also includes what it calls “complex multiple choice” items, which require 
students to answer a series of multiple-choice or true/false questions based on the same 
information. 

Both NAEP and PISA also include constructed-response items, for which students must supply the 
response. NAEP has “short answer” and “extended response” items. As described in the 2009 
NAEP reading framework: 

Short constructed-response items can be answered by one or two phrases or by one or two 
sentences; they should take students approximately 2 to 3 minutes to complete. Extended 
constructed-response items should elicit longer, more elaborated answers of a paragraph or 
two. They should take students approximately 5 minutes to complete. Scoring rubrics for 



10 This included 6 individual and 1 set of paired passages (8 total individual) from the NAEP 8th grade assessment; 
3 individual and 3 sets of paired passages (9 total individual) from the NAEP 12th grade assessment; and 21 
passages from the PISA assessment. 
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short and extended constructed-response items will focus on the content included in 
answers, not on spelling or grammatical considerations. However, students must answer 
constructed-response questions by using information from the text to receive credit 
(National Assessment Governing Board 2008, p. 40). 

PISA classifies its constructed-response items as open constructed response, short constructed 
response, and closed constructed response. Open constructed-response items may require a 
description or an explanation to support a response and may be scored for partial credit, though the 
acceptable length of response is much less than in NAEP. Short constructed-response items 
typically require students to supply a word or phrase or may require students to provide a specific 
response from the text. Closed constructed-response items are described as those that “require the 
student to generate a response, but that require minimal judgment on the part of a coder” (OECD 
2009, p. 46). 



Table A.4. Percentage distribution of items by item format in NAEP 2009 and PISA 2009 reading 
assessments 



Item Format 


NAEP 
grade 8 
(N =130) 


NAEP 
grade 12 
(N =131) 


PISA 
(N = 104) 


Multiple choice 


59 


58 


38 


Complex multiple choice 


t 


t 


9 


Constructed response 








Extended response 


10 


10 


t 


Open response 


t 


t 


35 


Short answer 


31 


32 


8 


Closed 


t 


t 


11 



t Not applicable. 

NOTE: Detail may notsum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2009 Reading 
Assessment: and Organization for Economic Cooperation and Development, Program for International Student Assessment (PISA) 2009 Reading Literacy 
Assessment. 



A.6.2 Cognitive Processes 



While the texts used form the content of the assessment, the cognitive processes (“aspects” in PISA 
and “cognitive targets” in NAEP) define the skills and abilities that students must draw on in 
response to the texts. Each item is written to primarily address one process. Although PISA and 
NAEP have three similarly named and defined cognitive process categories (shown in exhibit 3), 
there are differences that influence the kinds of items presented to students in each assessment. 
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Exhibit A.3 Cognitive process categories 



PISA 2009 


NAEP 2009 


Aspect categories 


Cognitive targets 


Access and 
retrieve 


Students navigate the 
information space provided 
to locate and retrieve one or 
more distinct pieces of 
information. 


Locate and recall 


Students locate or recall 
information from what they 
read; identify clearly stated 
main ideas or supporting 
details; and find essential 
elements of a story, such as 
characters, time, or setting. 


Integrate and 
interpret 


Students develop an 
understanding of the 
coherence of the text and 
make meaning from 
something that is not stated. 


Integrate and 
interpret 


Students integrate new 
information into their initial 
sense of what a passage says, 
often interpreting what they 
read in the process; make 
comparisons and contrasts 
of information or character 
actions; examine relations 
across aspects of text; and 
consider alternatives to what 
is presented in text. 


Reflect and 
evaluate 


Students are required to draw 
upon knowledge, ideas, or 
attitudes beyond the text in 
order to relate the 
information provided within 
the text to their own 
conceptual and experiential 
frames of reference. 


Critique and 
evaluate 


Students are required to 
stand back from what they 
read and view the text 
objectively. The focus 
remains on the text itself, but 
the reader’s purpose is to 
consider the text critically by 
assessing it from numerous 
perspectives and 
synthesizing what is read 
with other texts and other 
experiences. 



SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), PISA 2009 
Assessment Framework: Key Competencies in Reading, Mathematics and Science, 2009; and U.S. Department of Education, National Center for 
Education Statistics, National Assessment of Educational Progress (NAEP), Reading Framework for the 2009 National Assessment of Educational 
Progress, 2008. 

A.6.3 Experts’ Comparison of PISA and NAEP Reading Items 

The experts were also asked to review the items associated with the reviewed passages. They 
assessed the extent to which PISA and NAEP items fit into the other framework’s cognitive 
categories and whether or not the items fit the other framework in terms of the nature and format 
of the item. Additionally, items were reviewed for factors that contributed to their ease or difficulty. 
Panelists also looked at the PISA items to consider whether, in terms of level of challenge, they fit 
more closely to the NAEP 8*- or 12*-grade assessment. 
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A.6.4 Fit to Other Framework: Cognitive Processes 

In general, the NAEP and PISA items reviewed tended to fit within the other framework’s 
cognitive categories — that is, the items required a similar range and type of reading and thinking 
skills. Only about 4 to 6 percent of items in each assessment (including 8*- and 12th -grade NAEP 
and PISA) were rated as completely outside the other assessment’s framework cognitive categories. 
However, there were a number of items, especially in PISA, that were considered more borderline 
for cognitive fit or that were thought to fit within the other assessment’s framework, but in a 
different cognitive category . For example, there were some PISA items classified as “reflect and 
evaluate” that the panel thought would be considered “integrate and interpret” in the NAEP 
framework, as well as some PISA “integrate and interpret” items that might be classified as “locate 
and recall” in NAEP. Adjusting for these borderline items, the experts thought that, overall, about 
90 percent of both NAEP 8*- and 12*-grade items fit PISA’s cognitive categories tightly and well 
and that about 80 percent of PISA items fit the NAEP cognitive categories tightly and well. The 
expert panel remarked that it is more difficult for a student to read and answer questions from a 
passage that is significantly longer in length than it is from a shorter passage. The panel considered 
this difference when deciding the “fit” of PISA passages into the NAEP reading assessment 
cognitive categories. 

PISA items that were different from NAEP items cognitively included those that asked students to 
provide a personal stance or required a written response that was not dependent on text-based 
evidence and those that drew on multiple cognitive skills — scenarios that would not occur in 
NAEP. NAEP items that seemed different, or somewhat different, from PISA items cognitively 
were mainly NAEP vocabulary items, which required students to identify word meaning within the 
passage context and that have no corollaries in PISA. 

Within individual cognitive categories, the most challenging items to fit to the other framework’s 
cognitive categories appeared to be the PISA “reflect and evaluate” items. About 14 percent of 
these items did not match the cognitive categories of the NAEP framework at all, and about 20 
percent matched a different cognitive category in NAEP. PISA’s emphasis on inclusion of the 
student’s own experiences in “reflect and evaluate” items sometimes fit better NAEP’s “integrate 
and interpret” category than its “critique and evaluate” category. 

Fit to Framework: Item Format 

The experts found more differences between the assessments in item format than in cognitive skills 
measured. Although PISA and NAEP items tended to measure similar cognitive skills, they often 
were presented or formatted in ways that were dissimilar between the assessments. Over one-third 
of NAEP 8*-grade items and nearly two-fifths of NAEP 12*-grade items were judged incompatible 
with the PISA framework in terms of their nature or format; over half of the PISA items were 
judged incompatible with the NAEP framework. In general, the NAEP items that did not fit PISA 
were either vocabulary items or items that required a response that used information from the text 
to support it, for which PISA does not have corollary formats. PISA items did not fit NAEP for 
more varied reasons. Some PISA items did not fit NAEP because of the relatively short length of 
response acceptable for a correct answer, others because of the use of scaffolding or introduction to 
test items. Many items did not fit because of format differences. PISA used the closed constructed- 
response formats for low-level cognitive items in cases in which NAEP would only use multiple 
choice, and PISA used formats such as “complex multiple-choice” or included visuals in the item or 
responses, which also would not occur in the NAEP reading assessment. 
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A.6.5 Factors Contributing to Cognitive Challenge 

The experts reviewed the items to identify the factors that contributed to their cognitive challenge, 
or what drives item difficulty. Each item could be assigned multiple factors from a list of nine , the 
first 8 of which are based on factors described in the PISA framework (OECD 2009) and the 9* 
which was added by the experts: 

1. number of pieces of information needed to locate/ consider; 

2. amount of inference required; 

3. amount and prominence of competing information; 

4. length and complexity of text; 

5. type of interpretation required; 

6. familiarity with structure and genre; 

7. nature of knowledge needed to bring to item (narrow v. broad); 

8. depth of understanding required; and 

9. type of information. 

For both 8*- and 12*-grade NAEP, the most prevalent factors contributing to item challenge were 
the type of interpretation required by the test-taker, the number of pieces of information to be 
located or considered, and the depth of understanding required to answer the item correctly. Each 
of these factors was present in at least 35 percent of the items. In most cases, these were viewed as 
factors that contributed to increased item challenge — that is, the interpretations or understanding 
required were relatively complex or deep and the amount of information to be sorted through was 
relatively great. 

For PISA, the factors contributing to item challenge were typically the type of information that the 
student is required to handle, the number of pieces of information to be located or considered, and 
the amount of inference required. As in NAEP, these factors were present in at least 35 percent of 
items. Identifying the type of information that the student is required to handle was most often 
viewed as the factor that would increase the challenge level, because of PISA’s use of visual or 
graph-based information, which is not routinely found in NAEP and is considered fairly 
challenging. The other two factors were most often viewed as contributing to a relatively lower 
challenge level, as the amount of text to be sorted through was not as great as in NAEP and the 
amount of inference required was not as great as it might have been in the context of longer 
passages. 

Appropriate Level of Challenge for NAEP (PISA Items) 

In a final analysis, the experts examined whether or not the PISA items would be appropriate in 
terms of level of challenge for the NAEP 8*- or 12th-grade assessment. The experts considered 
what the PISA items required of students and how well that aligned with the items in the NAEP 
8*- and 12*-grade assessments. The experts found that about 55 percent of PISA items would be 
suitable for the NAEP 8*-grade assessment and about 15 percent would be suitable for the NAEP 
12th-grade assessment. However, about 30 percent of the items were thought to be inappropriate 
for NAEP in terms of level of challenge, and some of the items deemed suitable for the 8* grade 
were considered more borderline, or on the lower end of what would be acceptable, for this grade. 
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A.7 Summary 

NAEP measures in detail the reading knowledge of U.S. students as a whole, but can also provide 
trend information for individual states and some districts, different geographic regions, and 
demographic population groups. PISA provides a method for comparing the performance of U.S. 
students in reading with that of students in other nations. The two assessments differ in some key 
design elements. Differences include the following: 

• The content assessed by PISA and NAEP differ in subtle, but important ways. NAEP is 
tailored specifically to practices and standards used in the United States; in PISA, the 
content is determined internationally, in collaboration with other countries and reflecting 
consensus views of key content. Also, PISA’s specific focus on the “yield” of education 
systems and the application of competencies in real-world contexts distinguishes it from 
NAEP, which focuses more closely on measuring school-based performance. 

• Different target populations of students are assessed. Main NAEP uses grade-based 
samples targeting 4*-, 8*-, and 12*-grade students. PISA uses an age-based sample, which 
targets 1 5-year-olds, who are most likely between the ages of the NAEP target populations 
of 8*- and 12th-graders. 

• Measurement precision is greater in NAEP than in PISA. NAEP and PISA are both 
designed to provide valid and reliable measures of U.S. students’ performance in the 
aggregate as well as for major subpopulations, and each study draws a sample sufficient for 
this purpose. NAEP, however, is also designed to provide estimates for individual states, 
which requires an increased sample size, and thus measures performance at a higher level of 
precision than PISA. This difference may have an impact on the assessments’ sensitivities in 
detecting changes in student performance. 

• There is some overlap in how reading is defined in the two assessment programs and some 
similarities in how the frameworks are organized, with both NAEP and PISA specifying a 
cognitive dimension and a range of text types. However, there are subtle differences in how 
the cognitive categories are defined and more notable differences in the text types targeted 
for inclusion, as well as features (e.g., an assessment of vocabulary embedded within 
NAEP) that are unique to each assessment. 

• The passages selected for NAEP and PISA would likely fit in each other’s frameworks to 
only a limited degree. For example, NAEP passages, on average, are longer than PISA 
passages. Another, related difference is PISA’s more frequent use of graphic and other 
visual displays of text rather than continuous text passages. In terms of readability and 
grade level, PISA passages were generally more comparable to 12*-grade NAEP than to 8th- 
grade NAEP. 

• NAEP and PISA items generally tend to measure similar cognitive skills; however, they 
often are presented or formatted in ways that would not be interchangeable between the 
assessments. Key differences include PISA’s less extensive use of multiple-choice and more 
extensive use of short-constructed response formats than NAEP, while NAEP requires 
much longer, text-based responses for its extended constructed response formats. 

• Finally, there are differences in the source of challenge for NAEP and PISA items; these 
differences appear to be driven by the inclusion of longer passages in NAEP and the 
inclusion of more visual and other noncontinuous text formats in PISA. PISA items were 
found more frequently to be appropriate for the NAEP 8*-grade assessment than the 12th- 
grade assessment. 
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A.8 Contact Information 

Dan McGrath 

Director, International Activities Program 

National Center for Education Statistics 

U.S. Department of Education 

1990 K Street NW 

Washington, DC 20006 

Tel.: (202) 502-7426 

E-mail: Daniel.McGrath@ed.gov 

A.9 Useful Websites 



Eunice Greer 

National Assessment of Educational Progress 

National Center for Education Statistics 

U.S. Department of Education 

1990 K Street NW 

Washington, DC 20006 

Tel: (202) 502-7488 

E-mail: Eunice.Greer@ed.gov 



NAEP: http:/ /nces.ed.gov/nationsreportcard 

PISA: http:/ /www.nces.ed.gov/survevs/pisa (national) 

http:/ /www.pisa.oecd.org (international) 
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Table A.a. PISA and NAEP reading passage information by text format 





Continuous passages 


Noncontinuous passages 


NAEP 8 
(N =13) 


PISA 
(N = 14) 


NAEP 12 
(N = 11) 


NAEP 8 
(N =0) 


PISA 
(N = 15) 


NAEP 12 
IN =2) 


Length of passage and items per passage 


Averaqe number of words 


923.6 


385.9 


1127.7 


t 


325.0 


1425.5 


Ranqe of words in passaqes 


219-1,429 


115-758 


771 - 1,429 


t 


53 - 577 


1,262 - 1,589 


Average number of items 


10.0 


3.6 


10.0 


t 


3.6 


10.5 


Range of items 


9-11 


2-5 


9-11 


t 


2-5 


10- 11 



tNotapplicable, 

NOTE: NAEP 8 columns present averages and ranges of all reading passages categorized as grades 4/8, 8, and 8/12. NAEP 12 columns present 
averages and ranges of all reading passages categorized as grade 8/12 and 12. NAEP presents students with both individual passages and paired 
passages. Length analyses were calculated based on the passage or set of passages a student received with each set of items in orderto accurately 
reflect reading load. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2009 Reading 
Assessment: and Organization for Economic Cooperation and Development, Program for International Student Assessment (P ISA) 2009 Reading Literacy 
Assessment. 



Table A.b. PISA and NAEP readability analyses by text format 





Continuous passages 


Noncontinuous passages 


NAEP 8 
(N = 14) 


PISA 
(N = 10) 


NAEP 12 
(N = 13) 


NAEP 8 
(N=0) 


PISA 
(N =11) 


NAEP 12 
(N =3) 


Dale-Chall 


Average grade 
Range of grades 


6.9 


7.8 


7.0 


t 


7.4 


8.7 


5.2 -8.3 


6.1 -9.9 


5.4 -9.4 


t 


5.8 -8.3 


8.5 -9.1 


Flesch Reading Ease 


Average readability score 
Range of readability scores 
Flesch Readability identifier 


69.4 


58.1 


65.5 


t 


63.1 


48.7 


51-85 


32-88 


41-86 


t 


51-77 


42-56 


Standard 


Fairly difficult 


Standard 


t 


Standard 


Difficult 


Flesch-Kincaid Grade Level 


Average grade 
Range of grades 


7.5 


9.4 


8.7 


t 


8.1 


9.3 


4.8-11.5 


3.2-15.1 


5.3-13.4 


t 


4.8-10.6 


7.6-10.6 


FORCAST 














Average grade 
Range of grades 


9.5 


10.2 


9.4 


t 


10.1 


11.4 


7.9-10.6 


8.2-11.7 


8.3-10.8 


t 


9.3-11.4 


10.9-11.7 



tNotapplicable. 

NOTE: Only passages with word counts over 250 were included in the readability analyses. Excluded passages included three from NAEP and eight from 
PISA. NAEP 8 columns present averages and ranges of all individual reading passages categorized as grades 4/8, 8, and 8/12. NAEP 12 columns present 
averages and ranges of all individual reading passages categorized as grade 8/12 and 12. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2009 Reading 
Assessment: and Organization for Economic Cooperation and Development, Program for International Student Assessment (P ISA) 2009 Reading Literacy 
Assessment. 
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B.1 Letter to State Education Agencies from NCES Commissioner 




U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 

NATIONAL CENTER FOR EDUCATION STATISTICS 



[DATE] 

[NAME] 

[STATE DEPARTMENT OF EDUCATION] 

[ADDRESS] 

[CITY, STATE ZIP] 

Dear [CHIEF STATE SCHOOL OFFICER]: 

I am writing to inform you about the upcoming Program for International Student Assessment (PISA) 2009, in 
which the United States will participate along with more than 60 other countries. PISA provides international 
comparisons of student performance in reading, mathematics, and science literacy among 15-year -old students 
throughout the world. We are notifying you now because one or more schools in your state have been selected 
to take part in PISA 2009 in the fall of 2009. We ask your agency to support the participation of schools in 
your state in this study. 

PISA, described in more detail in materials enclosed with this letter, is sponsored in the United States by the 
National Center for Education Statistics in the U.S. Department of Education and conducted by Windwalker 
Corporation of McLean, VA and Westat of Rockville, MD. PISA has been endorsed by the Council of Chief 
State School Officers and a number of other national education organizations. The U.S. Office of Management 
and Budget has approved the data collection under OMB # 1850-0755. 

Within the next few days, a representative of Westat will contact sampled school districts and schools to 
discuss conducting data collection. In the meantime, if you have questions about the study, please feel free to 
call Dr. Paul Hopstock of Windwalker Corporation at (703) 970-3522 or send an email to 
PISA2009@westat.com. Also, more information about PISA is available at the NCES website at: 
http://nces.ed.gov/surveys/pisa/. 

Thank you for your support. 

Sincerely, 



Mark Schneider 
Commissioner 

cc: [STATE ASSESSMENT COORDINATOR] 
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B.2 Letter to School Districts from NCES Commissioner (Notification) 



U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 

NATIONAL CENTER FOR EDUCATION STATISTICS 



[SUPERINTENDENT NAME] 

[DISTRICT NAME] 

[ADDRESS] 

[CITY, STATE ZIP] 

Dear [SUPERINTENDENT]: 

I am writing to inform you about the upcoming Program for International Student Assessment (PISA) 2009, in 
which the United States will participate along with more than 60 other countries. PISA provides international 
comparisons of student performance in reading, mathematics, and science literacy among 15-year -old students 
throughout the world. We are notifying you now because one or more schools in your district have been 
selected to take part in PISA 2009 in the fall of 2009. 

We ask your agency to support the participation of schools in your district in PISA 2009. Since PISA is 
designed to test a representative sample of U.S. students, the accuracy of PISA depends on the full 
participation of the sampled schools and students. In appreciation for their time and efforts, the schools that 
participate will each receive an honorarium. In addition, school-level coordinators and participating students 
will also receive honoraria. 

PISA, described in more detail in materials enclosed with this letter, is sponsored in the United States by the 
National Center for Education Statistics in the U.S. Department of Education and conducted by Windwalker 
Corporation of McLean, VA and Westat of Rockville, MD. PISA has been endorsed by the Council of Chief 
State School Officers, the American Association of School Administrators, and a number of other national 
education organizations. The U.S. Office of Management and Budget has approved the data collection under 
OMB# 1850-0755. 

Within the next few days, a representative of Westat will contact the following school or schools in your 
district that have been selected for PISA 2009: [LIST ORIGINAL SCHOOLS HERE]. We may also contact as 
alternatives: [LIST REPLACEMENT SCHOOLS HERE] 

Information about districts, schools and students sampled for PISA 2009 is protected by the Education 
Sciences Reform Act of 2002 (PL 107-279). We will disclose the names of schools in each district only to the 
governing district for each school, and we ask that each district maintain the confidentiality of the sampled 
schools. Reports of the findings from PISA 2009 will not identify participating districts, schools, students, or 
individual staff. 

If you have any questions about PISA 2009 or your district’s participation, please feel free to call 1-888-270- 
6227 or send an email to PISA2009@westat.com. More information about PISA is also available at the NCES 
website at: http://nces.ed.gov/surveys/pisa/ . 
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Thank you for your support. 
Sincerely, 



Mark Schneider 
Commissioner 
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B.3 Letter to School Districts from NCES Commissioner (Consent) 




U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 

NATIONAL CENTER FOR EDUCATION STATISTICS 



[DATE] 

[SUPERINTENDENT NAME] 

[DISTRICT NAME] 

[ADDRESS] 

[CITY, STATE ZIP] 

Dear [SUPERINTENDENT]: 

I am writing to inform you about the upcoming Program for International Student Assessment (PISA) 2009, in 
which the United States will participate along with more than 60 other countries. PISA provides international 
comparisons of student performance in reading, mathematics, and science literacy among 15-year -old students 
throughout the world. We are notifying you now because one or more schools in your district have been 
selected to take part in PISA 2009 in the fall of 2009. 

The following school or schools in your district have been selected to take part in PISA 2009: [LIST 
ORIGINAL SCHOOLS HERE], We may also contact as alternatives: [LIST REPLACEMENT SCHOOLS 
HERE] 

We ask you to permit the participation of schools in your district in PISA 2009. Since PISA is designed to test 
a representative sample of U.S. students, the accuracy of PISA depends on the full participation of the sampled 
schools and students. In appreciation for their time and efforts, the schools that participate will each receive an 
honorarium. In addition, school-level coordinators and participating students will also receive honoraria. 

PISA, described in more detail in materials enclosed with this letter, is sponsored in the United States by the 
National Center for Education Statistics in the U.S. Department of Education and conducted by Windwalker 
Corporation of McLean, VA and Westat of Rockville, MD. PISA has been endorsed by the Council of Chief 
State School Officers, the American Association of School Administrators, and a number of other national 
education organizations. The U.S. Office of Management and Budget has approved the data collection under 
OMB# 1850-0755. 

Information about districts, schools and students sampled for PISA 2009 is protected by the Education 
Sciences Reform Act of 2002 (PL 107-279). We will disclose the names of schools in each district only to the 
governing district for each school, and we ask that each district maintain the confidentiality of the sampled 
schools. Reports of the findings from PISA 2009 will not identify participating districts, schools, students, or 
individual staff. 

Within the next few days, a representative of Westat will call you to discuss any questions that you may have 
and to discuss the procedures for obtaining your permission. In the meantime, if you have any questions about 
PISA 2009 or your district’s participation, please feel free to call 1-888-270-6227 or send an email to 
PISA2009@westat.com. More information about PISA is also available at the NCES website at: 
http://nces.ed.gov/surveys/pisa/ . 
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Thank you for your support. 
Sincerely, 



Mark Schneider 
Commissioner 
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B.4 Letter to Schools from NCES Commissioner 



U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 

NATIONAL CENTER FOR EDUCATION STATISTICS 

[DATE] 

[PRINCIPAL NAME] 

[SCHOOL NAME] 

[ADDRESS 1] 

[CITY, STATE ZIP] 

Dear [PRINCIPAL]: 

I am writing to inform you about the upcoming Program for International Student Assessment (PISA) 2009, in 
which the United States will participate along with more than 60 other countries. PISA provides international 
comparisons of student performance in reading, mathematics, and science literacy among 15-year-old students 
throughout the world. We are notifying you now because your school has been selected to take part in PISA 
2009 in the fall of 2009. 

I encourage your school’s participation in PISA 2009. In light of concerns about the nation’s international 
economic competitiveness and the skills of our workers, knowing how U.S. students compare with peers 
around the world is very important. This is why leading national education organizations, such as the National 
Association of Secondary School Principals, have endorsed the study. Your school can make a valuable 
contribution to learning more about where our education system has been successful and where we face 
challenges in educating our youth. 

Since PISA is designed to test a representative sample of U.S. students, the accuracy of PISA depends on the 
full participation of you and your students. In appreciation for your time and efforts, if your school 
participates, the school will receive a $200 honorarium, the school-level coordinator will receive $100, and 
participating students will receive $20 each. 

PISA, described in more detail in materials enclosed with this letter, is sponsored in the United States by the 
National Center for Education Statistics in the U.S. Department of Education and conducted by Windwalker 
Corporation of McLean, VA and Westat of Rockville, MD. The U.S. Office of Management and Budget has 
approved the data collection under OMB # 1850-0755. 

Information about districts, schools and students sampled for PISA 2009 is protected by the Education 
Sciences Reform Act of 2002 (PL 107-279). Reports based on the PISA 2009 data collection will not identify 
participating districts, schools, students, or individual staff. 

Within the next few days, a representative of Westat will call you to discuss any questions that you may have 
and to discuss the procedures for obtaining your permission. In the meantime, if you have any questions about 
PISA 2009 or your school’s participation, please feel free to call 1-888-270-6227 or send an email to 
PISA2009@westat.com. More information about PISA is also available at the NCES website at: 
http://nces.ed.gov/surveys/pisa/ . 
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Thank you for your support. 
Sincerely, 



Mark Schneider 
Commissioner 



2009 Data Files and Database with U.S. Specific Variables 



B-8 





B.5 Brochure 



Appendix B. School Recruitment Materials 



What countries are participating in 
PISA 2009? 



What organizations have endorsed 
PISA 2009? 



Europe 

United Kingdom 

Ireland 

Iceland 

Portugal 

Spain 

France 

Belgium 

Hie Netherlands 

Luxembourg 

Germany 

Denmark 

Norway 

Sweden 

Finland 

Switzerland 

Liechtenstein 

Austria 

Italy 

Slovenia 

Poland 

Czech Republic 
Slovak Republic 
Hungary 
Croatia 
Republic of 
Montenegro 
Republic or 
Serbia 



Greece 

Albania 

Bulgaria 

Romania 

Estonia 

Latvia 

Lithuania 

Moldova 

Azerbaijan 

Russian 

Federation 

North and 
South America 

Canada 
United States 
Mexico 
Dominican 
Republic 
Trinidad and 
Tobago 
Panama 
Colombia 
Brazil 
Uruguay 
Peru 

Argentina 

Chile 



Asia- Pacific 
and Africa 

Tunisia 

Turkey 

Israel 

Jordan 

Qatar 

Dubai (UAE) 
Kyrgyz Republic 
Kazakhstan 
Shanghai (China) 
HongKong- 
China 

Macao- China 

Singapore 

Chinese Taipei 

Thailand 

Korea 

Japan 

Indonesia 

Australia 

New Zeal and 




The following organizations support U.S. 
participation in the Program for International 
Student Assessment (PISA) 2009 as a means of 
assessing and comparing the reading > mathematics, 
and science skills of US. 15 -year old students with 
those of students from other countries. 

American Association of School Administrators 
American Federation of Teachers 
Council for American Private Education 
Council of Chief State School Officers 
Council of Great City Schools 
International Reading Association 
National Association of Secondary School Principals 
National Association of State Boards of Education 
National Association of Test Directors 
National Catholic Education Ass odati on. Department 
of Secondary Scho ol s 
National Christian School As sod ati on 
National Coundl of Teachers of Mathematics 
National Education Ass odati on 
National High School Association 
National Middle School Assodation 

How do I get more information? 

Visit the PISA website: 

http : //n ce s. e d.g ov/sur vey s/pi s a 

For questions about the PISA 2009» call the PISA 
Information Hotline at: 1-888-270-6227 

Or send an e-mail to: PISA2009@westat.com 





Program for 
International 
Student 

Assessment 

PISA 2009 
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Program for 
International 
Student Assessment 

PISA 2009 

What Is PISA? 

The Program for International Student 
Assessment (PISA) is an international 
assessment of 15 -year-olds. In 200V. 
students from more than 60 countries, 
including the United States, will participate. 
PISA measures how well students apply 
their knowledge and skills in trading, 
mathematics, and science literacy to 
problems within a real life context. 

PISA is sponsored by the Organization for 
Economic Cooperation and Development 
(OF.CD) and managed in the United States 
by the National Center for Education 
Statistics (NCES). part of the U.S. 
Department of Education. 

PISA has been conducted every 
three years since it began in 2000. 

PISA 2009 will include an m-depth 
assessment of reading, as well as 
shorter assessments of mathematics 
and science. In addition. PISA 2009 
will collect information on students' 
backgrounds, attitudes towards trading, 
and learning strategics. School principals 
will be asked to provide information on 
characteristics of the school and on its 
learning environment. 



Why Is PISA Important? 

In light of concerns about our international 
economic competitiveness and the skills of 
our workers, knowing how U.S. studenrs 
compare with peers around rhe world is 
imponant. PISA can show us where we 
hast been successful and where we lace 
challenges in educating our youth. 

PISA also compares US. student 
performance over time and across dilierent 
segments of the U.S. student population. 
PISA examines the relationships between 
student performance and classroom 
experiences. With these ourcomcs, PISA 



can suggest 
ways to improve 
the qualiryof 
educarion for US. 
students. 

What Is the 
nature of the 
assessment 
Items? 

Students will 
be asked to 
read various 
rypesof 

materials (for example, a scripr for a play, a 
chart or graph, a newspaper editorial, or a 
scientific text) and then answer questions 
about what they have read. The questions 
consist of a mix of multiple choice, short 
answer, and short essay questions. 

What will students be asked to do? 

Students will be asked to take pan in a 
3-hour session during which they will take 
thr assessment and complete a questionnaire 
about their backgrounds and artirudes 
toward reading. Studenrs will receive $20 
forpaniciparing. Participation is volunrary 
but important because each student 
represents many other students across the 
United States. A student's responses will be 
kept completely confidential, and individual 
responses will nor be presented in any report 
or publication. 
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Program tor International Student Assessment 

r PISA 200 9 

Summa ry of Activit ies for Schools 



When will data collection take place? 

I he assessment will take place during Ac tall of 2009. A 
PISA stall member will work with you t school to establish 
a convenient date tor the assessment. 

Who will be asked to take part in the 
assessment? 



PISA stall will work with wur school to select a random 
sample of up to 42 15-year-old students. Ihcsc students 
will be tested in a group. 



What is involved for the School Coordinator? 

Before the assessment, the School Coordinator will be 
asked to: 

• Provides list ul all 15-year-okl students in die school: 

• Review parent notification requirements: 

• Work with PISA sulf to schedule the assessment and 
secure a large, quiet space for testing (space tor up to 
42 students); 



Who conducts the assessment? 

Trained PISA stall will administer Ac assessment and 
provide all the required materials Tlrus. teachers will not 
need to set aside time for administering Ac assessment. 

Will the school, staff members, and students 
receive compensation for their time and 
effort? 

Yes. For yutir participation in PISA 2009, jour school 
will receive $200. Also, school principals will he asked 
to designate a School Coordinator (such as a school 
counselor or teacher) to work wiA PISA stall on wiAin- 
school arrangements. The School Coordinator will receive 
S100. Each student who participates in the assessment will 
receive $20, if such payments are approved by the school 
principal. 

What is Involved for school staff and 
students? 

• Schools will be asked to designate a School 
Coordinator toworkwith PISA station wirhin- 
school arrangements. 

• Ihc school principal or a designee will be asked to 
complete a JO-minutr school questionnaire that 
asks about characteristics ol the school and its 
environment. 

• Students will he asked to take part in an assessment 
that is divided into two par ts. 1 he first part is a 
reading, mathematics, and science assessment that 
will Last about 2 hours and is separated into two 
sessions with a lueak in the middle. After another 
break, students will be asked to complete a Student 
Questionnaire, which will rake about 30 minutes. 



« Work with PISA staff to identify students with special 
needs who will not he asked to participate in Ac 
assessment; 

• Notify parents, teachers, and students concerning the 
Students who have been selected to lake pari and the 
benefits of participation ; 

« Confirm Ac final list of Ac sampled students; 

• Confirm Ac date and location tor Ac testing. 

« Receive the school questionnaire and distribute and 
collect it from Ac principal or designee: and 

• Remind the students of Ae time and place of Ae 
assessment and urge them to attend. 

On assessment rLiy, Ae School Coord ina tor wi 1 1 be asked 
tD: 

• fin sure that Ac testing room is available and set up tor 
testing; and 

• Ptovide assistance to ensure that selected students 
attend the assessment session. 

PISA requites very high student participation rates, and 
data from individual schools will not be counted unless 
most of the selected students attend the assessment. Ihus 
it is extremely important rhat attendance rates are high 
in order to avoid the need for a makeup session. 
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More Information on PISA 

What is PISA? 

PISA is an international assessment designed by 
participating countries and administered to 1 5 -year -old 
students in schools around the world. PISA reports on 
performance in reading .mathematics, and sdence literacy 
every three years, with emphasis on one subject in each data 
collection cyde. PISA began in 2000 focusing on reading 
literacy. PISA focused on mathematics literacy in 2003 and 
science literacy in 2006. In 2009, reading literacy again 
will be theprimaryfocus. Educators throughout the U.S. 
ben efit fr om P ISA by hating agreaterunderstandingof 
how 15-year-olds in the U.S. compare in terms ofreading, 
mathematics, and science knowledge and skills with similar 
stu dent s worl dwi de . 

PISA is sponsore d by the Organization for Economic 
Cooperation and Development (OECD) andmanagedin 
the United States by the National Center for Education 
Statistics (NCES),part of the US. Department of 
Education. 

Do schools and students have to participate? 

Although participation is voluntary, it is important that 
every selected school and student participate to ensure 
the completeness and accuracy of results. Each school and 
student represents many others. Thus, the validity of results 
dep end s up on a high p artidpati on r ate . 




What confidentiality protections are included in 
the assessment? 



The list of schools and students included in PISA 2009 
will be confidential. Student names will not be linked to 
assessment results or questionnaire responses on any data 
files (unique ID numbers will be used), and individual 
student scores or responses will never be reported. 

Ifyouhave any questions aboutthe PISA 2009, please feel 
free to contact the PISA staff at PISA2009@westat.com or 
on the toll-free number at 1-888-270-6227. 
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Student and Parent Materials 
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Appendix C. Student and Parent Materials 



C.1 Student Invitation 





Welcome to 
the U.S. PISA Team! 

You have been selected to be part of a small group 
of U.S. students representing the United States in the 

Program for International Student Assessment (PISA) 2009 

This fall students in the United States will match their 
knowledge and skills in reading, mathematics and science 
against students from 64 other countries. 

When? Where ? What time ? 

PISA Date: 

PISA Time: 

PISA Location: 
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Appendix C. Student and Parent Materials 



C.2 Explicit Parent Materials 

Explicit Consent Letter 



Dear Parent or Guardian, 

This letter is to ask you to allow your child to take part in an important international 
study of student learning. The study is known as PISA (the full name is Program for 
International Student Assessment). PISA looks at student learning in reading, mathematics, and 
science around the world. It has been conducted every three years since 2000, and documents 
world-wide trends in the knowledge and skills of 15-year olds in these areas. Along with 66 
other nations, the United States will take part in PISA in 2009, just as we have in previous years. 

Your child’s school has accepted an invitation to take part in PISA 2009, and your child 
is one of approximately 40 students who have been invited to take part. It is important that 
each student selected take part in the assessment. Each time a selected student does not 
participate, the accuracy of the U.S. information suffers. I urge you to support this effort 
by allowing and encouraging your child to take part. The enclosed summary sheet provides 
some background information on PISA, explains what is involved for each student, and gives a 
contact phone number and e-mail address where you can get answers to any questions you might 
have. 



For participating in PISA, the school is receiving a $200 honorarium. Each of the 
participating students also will receive $20 as a token of appreciation. 

It goes without saying that all of the information collected is completely confidential. In 
fact, the study is required to do so by law. So, students and schools are never identified in any 
reports. 



Before your child can take part in the PISA assessment, the study must have your 
written consent. You can let us know by completing the attached form and returning it to 
the school. The form should be returned to [School Coordinator] by [Date]. 

Thank you for taking the time to think about this study. 
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Appendix C. Student and Parent Materials 



Explicit Consent Form 

Program for International Student Assessment (PISA 2009) 
Parent/Guardian Consent Form 

Your child has been asked to take part in an international study of student learning called 
the Program for International Student Assessment (PISA). This assessment will be conducted by 
a team of researchers from Westat Corporation, who are operating under contract with 
Windwalker Corporation on behalf of the U. S. Department of Education. In the Fall of 2009, 
the assessment will be administered in approximately 160 schools and with approximately 5,000 
students in the United States. The study needs your permission for your child to take part. 

Confidentiality 

Every precaution will be taken to protect your child’s privacy. Your child’s name will 
never be associated with any results that are reported from the assessment. Only group-level 
findings will be presented in any reports or other documents that are created as part of the 
assessment. No individually identifiable results will be presented. 

Your Child’s Rights 

1. Your child’s participation in the assessment is completely voluntary. However, we 
encourage each student selected take part. Each time a selected student does not 
participate, the accuracy of the U.S. information suffers. Refusal to participate will 
not affect your child’s opportunities in school. You or your child may decide not to 
participate. 

2. The assessment will last for about 3 hours. 

3. Your child will be given a $20 check at the completion of the assessment in appreciation 
for his or her contribution. 

4. If you have any questions about PISA or your child’s rights as a study participant, you 
can contact PISA at 1-888-270-6227 or PISA2009@westat.com. 

5. You will be given a copy of this form to keep. 



Signature of Parent: 

Name of Student: 

Street Address 

City Zip code 

Telephone 

Date 



FOR OFFICE USE ONEY: 
Student ID: 
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Appendix C. Student and Parent Materials 



C.3 Implicit Parent Materials 

Implicit Consent Letter 



Dear Parent or Guardian, 

This letter is to inform you about an important international study of student learning 
being conducted in your child’s school. The study is known as PISA (the full name is Program 
for International Student Assessment). PISA looks at student learning in reading, mathematics, 
and science around the world. It has been conducted every three years since 2000, and 
documents world-wide trends in the knowledge and skills of 15-year olds in these areas. Along 
with 66 other nations, the United States will take part in PISA in 2009, just as we have in 
previous years. 

Your child’s school has accepted an invitation to take part in PISA 2009, and your child 
is one of approximately 40 students who have been invited to take part. It is important that 
each student selected take part in the assessment. Each time a selected student does not 
participate, the accuracy of the U.S. information suffers. I urge you to support this effort 
by allowing and encouraging your child to take part. The enclosed summary sheet provides 
some background information on PISA, explains what is involved for each student, and gives a 
contact phone number and e-mail address where you can get answers to any questions you might 
have. 



For participating in PISA, the school is receiving a $200 honorarium. Each of the 
participating students also will receive $20 as a token of appreciation. 

It goes without saying that all of the information collected is completely confidential. In 
fact, the study is required to do so by law. So, students and schools are never identified in any 
reports. 



If you are willing to allow your teenager to participate, you do not need to return 
the attached form. If for any reason you object to your teenager’s participation, please fill 
out the enclosed form and return it to [School Coordinator] at the school by [Date]. 

Thank you for taking the time to think about this study. 
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Implicit Consent Form 

Program for International Student Assessment (PISA 2009) 

Parent Permission Form 

Your child has been asked to participate in an international study of student learning called the 
Program for International Student Assessment (PISA). This assessment will be conducted by a 
team of researchers from Westat Corporation, who are operating under contract with 
Windwalker Corporation on behalf of the U. S. Department of Education. In the Fall of 2009, 
the assessment will be administered in approximately 160 schools and with approximately 5,000 
students in the United States. 

IF YOU GRANT YOUR PERMISSION FOR YOUR TEENAGER TO PARTICIPATE IN 
PISA 2009, YOU DO NOT NEED TO RETURN THIS FORM. 



IF YOU DO NOT CONSENT TO YOUR CHILD’S PARTICIPATION IN PISA 2009, 
PLEASE RETURN THIS FORM TO YOUR TEENAGER’S SCHOOL AS SOON AS 
POSSIBLE. 

I DO NOT GRANT PERMISSION for my child, _, to 

participate in the Program for International Student Assessment 2009. 



(Signature of parent or guardian) 



Date of signature: / / 



( .). 

Area code Telephone number 

PLEASE PRINT: 

Student name: 



School Name: 



FOR OFFICE USE ONLY: 
Student ID: 
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Appendix C. Student and Parent Materials 



C.4 Parent Notification 



Notification Letter 



Dear Parent or Guardian, 

This letter is to inform you about an important international study of student learning 
being conducted in your child’s school. The study is known as PISA (the full name is Program 
for International Student Assessment). PISA looks at student learning in reading, mathematics, 
and science around the world. It has been conducted every three years since 2000, and 
documents world- wide trends in the knowledge and skills of 15-year-olds in these areas. Along 
with 66 other nations, the United States will take part in PISA in 2009, just as we have in 
previous years. 

Your child’s school has accepted an invitation to take part in PISA 2009, and your child 
is one of approximately 40 students who have been invited to take part. It is important that 
each student selected take part in the assessment. Each time a selected student does not 
participate, the accuracy of the U.S. information suffers. I urge you to support this effort 
by encouraging your child to take part. The enclosed summary sheet provides some 
background information on PISA, explains what is involved for each student, and gives a contact 
phone number and e-mail address where you can get answers to any questions you might have. 

For participating in PISA, the school is receiving a $200 honorarium. Each of the 
participating students also will receive $20 as a token of appreciation. 

It goes without saying that all of the information collected is completely confidential. In 
fact, the study is required to do so by law. So, students and schools are never identified in any 
reports. 

Thank you for taking the time to think about this study. 
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Appendix C. Student and Parent Materials 



C.5 Fact Sheet for Parents 



Facts for Parents 
About PISA 2009 



In the fall of this year, your child’s school will be one of about 160 nationwide taking part in 
PISA 2009. The schools were selected randomly to represent the nation’s schools, and within 
each school, about 40 students were selected randomly to take part. Your child was among the 
students selected to take part in the study. 

What is PISA? 

PISA (the Program for International Student Assessment) is an international assessment that 
measures student learning in reading, mathematics, and science. The assessment occurs every 
three years (2000, 2003, 2006 and 2009), and provides information about how students in the 
U.S. compare in achievement with students in other countries. Sixty-seven (67) countries will be 
participating in PISA 2009. The National Center for Education Statistics within the U.S. 
Department of Education sponsors U.S. participation in PISA. 

What is involved? 

PISA staff will visit the school and administer a two-hour assessment to the selected students. 
There is one break during the assessment. Students will also be asked to complete a background 
questionnaire that takes about 30 minutes to complete. 

What are the benefits? 

The nation as a whole benefits from PISA by having a greater understanding of how the 
knowledge and skills of U.S. students compare with those of students from other countries. 
Schools that participate in PISA will receive $200, and each student who participates will receive 
a $20 check. 

Where can I find out more about PISA? 

More information about PISA is available at the PISA website at 

http ://w w w .nces .ed.gov/surveys/pisa. If you have specific questions you can call PISA staff at 
1-888-270-6227 or e-mail us at PISA2009@westat.com. 
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PISA 2009 U.S. Student Questionnaire 

and School Questionnaire 
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ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT 



D.1 Student Questionnaire 




OECD Program for 
International Student Assessment 2009 



OECD 
PIS. A 



Core B Consortium 

Crto Institute for Educational Measurement 
University of Twente 

University of Jyvaskyla, Institute for Educational Research 
Direction de I' Evaluation de la Prospective et de la Performance (DEPP) 



Le 



ng 



Snides! ID 

ParocipatiocL Status 

Enghsh iu 



Date of Test 



USA 

Main Stmty PISA 2009 

/ 1 2009 



Month Day 



STUDENT QUESTIONNAIRE 



Place Label Here 
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In this booklet you will find questions about: 



■ You 

■ Your family and your home 

■ Your reading activities 

■ Learning time 

■ Classroom and school climate 

■ Your English classes 

■ Libraries 

■ Your strategies in reading and understanding texts 

In some of the questions you will be asked about reading. What we 
specifically mean by reading is the skill to understand, use and think about 
written texts. This skill is needed to reach one’s goals, to develop one's 
knowledge and potential, and to take part in society. 

Please read each question carefully and answer as accurately as you can. In 
the test you usually circled your answers. For this questionnaire, you will 
normally answer by darkening a circle. For a few questions you will need to 
write a short answer. 

If you make a mistake when darkening a circle, erase your mistake and darken 
the correct circle. If you make a mistake when writing an answer, simply 
cross it out and write the correct answer next to it. 

In this questionnaire, there are no right or wrong answers. Your answers 
should be the ones that are right for you. 

You may ask for help if you do not understand something or are not sure how 
to answer a question. 

Your answers will be combined with others to make totals and averages 
in which no individual can be identified. All your answers will be kept 
confidential. 
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SECTION 1: ABOUT YOU 



Q1 What grade are you in? 



grade 



Q2 How long have you been in this school? 

( Please darken only one circle.) 



Less than one year f ) 

One to two years Q 

Three to four years O, 

More than four years (j 4 



Q3 When were you born? 

(Please write the month, day and year you were born ) 

19 

Month Day Year 



Q4 Are you female or male? 

Female Male 

Q, o 2 



2009 Data Files and Database with U.S. Specific Variables 



D-4 





Q5 Which best describes you? 

( Please darken only one circle.) 

I am Hispanic or Latino. o, 

I am not Hispanic or Latino. O ' 2 

Q6 Which of these categories best describes your race? 

( Please darken one or more circles.) 



White q 

Black or African American Q 

Asian Q 

American Indian or Alaska Native Q 

Native Hawaiian or Other Pacific Islander Q 
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Q7 



Did you attend pre-school? 



No 




Yes, for one year or less 


q 2 


Yes, for more than one year 





Q8 Did you attend kindergarten? 



No 


o a 


Yes 


o 2 



Q9 How old were you when you started first grade? 

years 

Q10 Have you ever repeated a grade? 

( Please darken only one circle in each row) 





No, never 


Yes, once 


Yes , twice or 
more 


a) In kindergarten 


Of 


o 2 


Q'3 


b) In grades 1-6 


it 


0 2 


o 3 


c) In grades 7-9 




o 2 


o 3 
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d) In grades 10-12 



Oj Q\ Q 3 



Q1 1 What is the highest grade or level of school you expect 
to complete? 

( Please darken only one circle.) 

Less than high school Q 

High school O , 

Vocational or technical certificate (such as 
cosmetology or auto mechanics) 3 

Associate’s degree (2-year degree from a 
community college) 4 



Bachelor’s degree (4-year college degree) 

Master’s degree Q 

Doctoral or professional degree such as 
medicine or law 7 
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SECTION 2: YOUR FAMILY AND YOUR HOME 



In this section you will be asked some questions about your family and your home. 

Some of the following questions are about your mother and father or those persons 
who are like a mother or father to you — for example , guardians, step-parents, 
foster parents, etc. 

If you share your time with more than one set of parents or guardians, please 
answer the following questions for those parents/guardians you spend the most 
time with. 

Q12 Who usually lives at home with you? 

( Please darken one circle in each row) 





Yes 


No 


a) Mother (including stepmother or foster mother) 


€>i 


o 2 


b) Father (including stepfather or foster father) 


Or 


o 2 


c) Brother(s) (including stepbrothers) 


Or 


o 2 


d) Sister(s) (including stepsisters) 


o, 


o 2 


e) Grandparent(s) 


0, 


o 2 


f) Others (e.g., cousin) 


O t 


o 2 
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Q13a What is your mother’s main job? 

(e.g., school teacher, cook, sales manager) 

(If she is not working now , please tell us her last main job) 

Please write in the job title. 

Q13b What does your mother do in her main job? 

(e.g., teaches high school students, helps prepare meals in a 
restaurant, manages a sales team) 

Please use a sentence to describe the kind of work she does or did in that 
job. 



Q14 What is the highest level of schooling (not including college) 
completed by your mother? 

If you are not sure which circle to choose, please ask the test administrator 
for help. 

( Please darken only one circle) 



She completed grade 12 (high school 


o t 


diploma or GED). 


She completed grade 9. 


o 2 


She completed grade 6. 


0 3 


She did not complete grade 6. 


■o 4 
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Q15 Does your mother have any of the following degrees, 
certificates or diplomas? 

If you are not sure how to answer this question, please ask the test 
administrator for help. 

( Please darken one circle in each row) 





Yes 


No 


a) Master’s, doctoral, or professional degree 
such as medicine or law 


C\ 


o 2 


b) Bachelor’s degree (4-year college degree) 


O, 


o 2 


c) Associate’s degree (2-year degree from a 
community college) 


o, 


o 2 


d) Vocational or technical certificate/diploma 
after high school (such as cosmetology or 


% 


o 2 



auto mechanics) 



Q16 What is your mother currently doing? 

( Please darken only one circle) 



Working full-time for pay Q 

Working part-time for pay 0 2 

Not working, but looking for a job 0 3 

Other (e.g., home duties, retired) 0 4 
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Q17a What is your father’s main job? 

(e.g., school teacher, cook, sales manager) 

(If he is not working now, please tell us his last main job) 

Please write in the job title. 

Q17b What does your father do in his main job? 

(e.g., teaches high school students, helps prepare meals in a 
restaurant, manages a sales team) 

Please use a sentence to describe the kind of work he does or did in that 
job. 



Q18 What is the highest level of schooling (not including college) 
completed by your father? 

If you are not sure how to answer this question, please ask the test 
administrator for help. 

(Please darken only one circle) 



He completed grade 12 (high school 

diploma or GED). ^ 1 

He completed grade 9. O, 

He completed grade 6. 0 3 

He did not complete grade 6. 0 4 
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Q19 Does your father have any of the following degrees, certificates 
or diplomas? 

If you are not sure which circle to choose, please ask the test administrator 
for help. 

( Please darken one circle in each row) 





Yes 


No 


a) Master’s, doctoral, or professional degree 
such as medicine or law 


o x 


o 2 


b) Bachelor’s degree (4-year college degree) 


O, 


:o 2 


c) Associate’s degree (2-year degree from a 
community college) 


O, 


o 2 


d) Vocational or technical certificate/diploma 
after high school (such as cosmetology or 


O, 


o 2 



auto mechanics) 



Q20 What is your father currently doing? 

( Please darken only one circle) 



Working full-time for pay Q 

Working part-time for pay 0 2 

Not working, but looking for a job 0 3 

Other (e.g., home duties, retired) 0 4 
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Q21 In what country were you and your parents born? 

( Please darken one circle in each column ) 





You 


Mother 


Father 


United States* 


O 0 i 


O 0 . 


o„i 


Other country 


O 02 


a* 


O 02 



*NOTE: the “United States” refers to the 50 states. District of Columbia, and U.S. military bases abroad. 

Q22 If you were NOT born in the United States, how old were you 
when you arrived in the United States? 

If you were less than 12 months old, please write zero (0). 

If you were born in the United States please skip this question and go to 
Q23. 

years 

Q23 What language do you speak at home most of the time? 

( Please darken only one circle) 

English 0 313 

Spanish ( 3 156 

Other language 0 859 
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Q24 Which of the following are in your home? 

( Please darken one circle in each row ) 








Yes 


No 




a) A desk to study at 


o, 


q 2 




b) A room of your own 


0, 


o 2 




c) A quiet place to study 


O, 


o 2 




d) A computer you can use for school work 


O, 


o 2 




e) Educational software 


o, 


0 2 




f) A link to the Internet 


o, 


o 2 




g) Classic literature (e.g., Shakespeare) 


o, 


q 2 




h) Books of poetry 


o* 


o 2 




i) Works of art (e.g., paintings) 


o, 


o 2 




j) Books to help with your school work 


o, 


o 2 




k) Technical reference books or manuals 


O, 


0 2 




1) A dictionary 


C, 


g 2 




m) A dishwasher 


O, 


o 2 




n) A DVD player 


A 


o 2 




o) A guest room 


o 3 


o 2 




p) A high-speed Internet connection 


o, 


o 2 
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q) A musical instrument 



, o 2 



Q25 How many of these are there at your home? 

( Please darken only one circle in each row) 

Three or 





None 


One 


Two 


more 


a) Cell phones 


0 


o 


o 


g 4 


b) Televisions 


o 


0 


0 


o 4 


c) Computers 


o 


o 


o 


o 4 


d) Cars 


0 


o 


o 


o 4 


e) Bathrooms with a bathtub or shower 


o 


o 


o 


0 4 
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Q26 How many books are there in your home? 

There are usually about 15 books per foot of shelving. Do not include 
magazines, newspapers, or your schoolbooks. 

(Please darken only one circle) 



0-10 books Qj 

11-25 books O 

w 2 

26-100 books GL 

w 3 

101-200 books 0* 

201-500 books 0 5 

More than 500 books 

6 
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SECTION 3: YOUR READING ACTIVITIES 

The questions in this section are mainly about your reading activities outside 
school. 

Q27 About how much time do you usually spend reading for 
enjoyment? 

( Please darken only one circle) 

I do not read for enjoyment. Q 

30 minutes or less a day Q 

More than 30 minutes to less than 
60 minutes a day ^ 3 

1 to 2 hours a day Q 

More than 2 hours a day Q 
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Q28 How much do you agree or disagree with these statements 
about reading? 

( Please darken only one circle in each row) 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


a) I read only if I have to. 


C\ 


o 2 


O, 


o 4 


b) Reading is one of my favorite hobbies. 


o, 


o 2 


o 3 


o 4 


c) I like talking about books with other people. 


C\ 


o 2 


o 3 


C> 4 


d) I find it hard to finish books. 


o t 


o 2 


o 3 


o 4 


e) I feel happy if I receive a book as a present. 


C\ 


o 2 


o 3 


o 4 


f) For me, reading is a waste of time. 


C' : 1 


o 2 


o 3 


o 4 


g) I enjoy going to a bookstore or a library. 


0, 


o 2 


o 3 


o 4 


h) I read only to get information that I need. 


C\ 


o 2 


o 3 


o 4 


i) I cannot sit still and read for more than a few 
minutes. 


o i 


o 2 


o 3 


0 4 


j) I like to express my opinions about books I 
have read. 


C\ 


o 2 


o 3 


o 4 


k) I like to exchange books with my friends. 


C\ 


o 2 


o 3 


q 4 
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Q29 How often do you read these materials because you want to ? 



( Please darken only one circle in each row) 





Never or 
almost 
never 


A few 
times a 
year 


About 
once a 
month 


Several 
times a 
month 


Several 
times a 
week 


a) Magazines 


Q.t 


o 2 


0 3 


o 4 


o 5 


b) Comic books 


o t 


o 2 


0.3 


o 4 


o 5 


c) Fiction (e.g., novels, narratives, 
stories) 


O, 


0 2 


O3 


o 4 


o 5 


d) Non-fiction books 


o, 


o 2 


0 3 


o 4 


o 5 


e) Newspapers 


©. 


o 2 


O3 


o 4 


o 5 
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Q30 How often are you involved in the following reading activities? 

( Please darken only one circle in each row. If you don’t know what the 
activity is, darken “I don ’t know what it is. ”) 





I don’t 
know 
what it is 


Never or 
almost 
never 


Several 
times a 
month 


Several 
times a 
week 


Several 
times a 
day 


a) Reading emails 


Q* 


o 2 


0 3 


o 4 


o 5 


b) Chatting on line 


o, 


o a 


O3 


o 4 


G 5 


c) Reading online news 


o, 


G 2 


0, 


o 4 


o 5 


d) Using an online dictionary or 
encyclopedia (e.g., Wikipedia®) 


Q. 


o 2 


o 3 


o 4 


o 5 


e) Searching online information to 
learn about a particular topic 


o, 


0 2 


0 3 


o 4 


o 5 


f) Taking part in online group 
discussions or forums 


Q, 


o 2 


o 3 


o 4 


o 5 


g) Searching for practical 
information on line (e.g., 
schedules, events, tips, recipes) 


0, 


o 2 


O3 


Q, 


o 5 


h) Text-messaging 


O, 


o 2 


O 3 


o 4 


0 5 
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Q31 When you are studying, how often do you do the following? 

( Please darken only one circle in each row) 

Almost Almost 



a) When I study, I try to memorize everything 
that is covered in the text. 


never 

C if 


Sometimes 

o 2 


Often 

o 3 


always 

o 4 


b) When I study, I start by figuring out what 
exactly I need to learn. 




0 2 


O, 


o 4 


c) When I study, I try to memorize as many 
details as possible. 


O, 


4 


o 3 


o 4 


d) When I study, I try to relate new information 
to prior knowledge acquired in other 
subjects. 


0t 


o 2 


q 3 


0 4 


e) When I study, I read the text so many times 
that I can recite it. 


a. 


0.2 


o 3 


o 4 


f) When I study, I check if I understand what I 
have read. 


Ox 


-o 2 


q 3 


o 4 


g) When I study, I read the text over and over 
again. 


o, 


0 2 


0 3 


o 4 


h) When I study, I figure out how the 

information might be useful outside school. 


c\ 


0 2 


o 3 


o 4 


i) When I study, I try to figure out which 
concepts I still haven’t really understood. 


O, 


o 2 


0 3 


o 4 


j) When I study, I try to understand the 
material better by relating it to my own 
experiences. 


O, 


o 2 


O, 


0 4 


k) When I study, I make sure that I remember 
the most important points in the text. 


O, 


o 2 


o 3 


o 4 


1) When I study, I figure out how the text 
information fits in with what happens in real 
life. 


o $ 


0 2 


0 3 


0 4 


m) When I study and I don’t understand 

something, I look for additional information 


O, 


q 2 


o 3 


o 4 



to clarify this. 
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SECTION 4: LEARNING TIME 



Q32 How many minutes, on average, are there in a class period for 
the following subjects? 

Minutes in a class period in English (English classes may 

include those in literature, creative writing, journalism, Minutes 

etc.): 

Minutes in a class period in mathematics: Minutes 



Minutes in a class period in science: 



Minutes 



Q33 How many class periods per week do you typically have for the 



following subjects? 

Number of class periods per week in English: 


class periods 


Number of class periods per week in mathematics: 


class periods 


Number of class periods per week in science: 


class periods 



Q34 In a normal, full week at school, how many class periods do you 
have in total? 

Number of ALL class periods: class periods 
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Q35 What type of out-of-school-time lessons do you attend 
currently? 

These are only lessons in subjects that you are also learning at school, 

that you spend extra time learning outside of normal school hours. The 
lessons may be given at your school, at your home or somewhere else. 

(Please darken only one circle in each row) 





Yes 


No 


a) Enrichment lessons in English 




o 2 


b) Enrichment lessons in mathematics 


o x 


o 2 


c) Enrichment lessons in science 


o x 


a* 


d) Enrichment lessons in other school subjects 


Pi 


o 2 


e) Remedial lessons in English 


o x 


o 2 


f) Remedial lessons in mathematics 


Qi 


o 2 


g) Remedial lessons in science 


o x 


o 2 


h) Remedial lessons in other school subjects 


•o x 


o 2 


i) Lessons to improve your study skills 


o x 


o 2 
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Q36 How many hours do you typically spend per week attending out- 
of-school-time lessons in the following subjects (at school, at 
home or somewhere else)? 

These are only lessons in subjects that you are also learning at school, that 
you spend extra time learning outside of normal school hours. The lessons 
may be given at your school, at your home or somewhere else. 

(Please darken one circle in each column ) 





English 


Mathematics 


Science 


Other subjects 


I do not attend out-of- 
school-time lessons in these 
subjects. 


o, 


0 , 


o t 


Qt 


Less than 2 hours a week 


o 2 


o 2 


o 2 


o 2 


2 or more but less than 4 
hours a week 


o 3 


0 3 


o 3 


o 3 


4 or more but less than 6 
hours a week 


o 4 


o 4 


o 4 


O, 


6 or more hours a week 


o 5 


o 5 


o 5 


o 5 
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SECTION 5: YOUR SCHOOL 



Q37 Thinking about what you have learned in school: To what 
extent do you agree or disagree with the following 
statements? 

( Please darken only one circle in each row) 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


a) School has done little to prepare me 
for adult life when I leave school. 


o x 


o 2 


0 3 


o 4 


b) School has been a waste of time. 




G 2 


0 3 


o 4 


c) School has helped give me 
confidence to make decisions. 


O, 


o 2 


0 3 


o 4 


d) School has taught me things which 
could be useful in a job. 


O, 


o 2 


0 3 


q 4 
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Q38 How much do you disagree or agree with each of the 
following statements about teachers at your school? 

( Please darken only one circle in each row.) 



a) I get along well with most of my 
teachers. 

b) Most of my teachers are interested in 
my well-being. 

c) Most of my teachers really listen to 
what I have to say. 

d) If I need extra help, I will receive it 
from my teachers. 

e) Most of my teachers treat me fairly. 



Strongly Strongly 

disagree Disagree Agree agree 

Oj 0 2 ©3 ©4 

0* o 2 o 3 o 4 

© i o 2 o 3 © 4 

©1 ©2 ©3 ©4 

Ox ©2 ©3 ©4 
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SECTION 6: YOUR ENGLISH CLASSES 



Q39 On average, about how many students attend your 
English class(es)? 

students 



Q40 How often do these things happen in your English 
classes? 

( Please darken only one circle in each row) 



Never 





or 


In 








hardly 


some 


In most 


In all 




ever 


classes 


classes 


classes 


a) Students don’t listen to what the teacher 
says. 


Q. 


o 2 


0 3 


o 4 


b) There is noise and disorder. 


O, 


o 2 


0 3 


o 4 


c) The teacher has to wait a long time for 


O, 


o 2 


0 3 


q 4 


the students to quiet down. 


d) Students cannot work well. 


O, 


o 2 


0 3 


o 4 


e) Students don’t start working for a long 
time after the class begins. 


O, 


o 2 


0 3 


o 4 
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e) The teacher encourages students to express 
their opinions about a text. 



f) The teacher helps students relate the stories 
they read to their lives. 



g) The teacher shows students how the 

information in texts builds on what they c>i Q 2 0 3 U 4 

already know. 
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Q42 



a) 



b) 



c) 



d) 



e) 



f) 

g) 

h) 



i) 



In your English classes, how often does the following occur? 

(Please darken only one circle in each row ) 





Never 

or 

hardly 

ever 


In some 
classes 


In most 
classes 


In all 
classes 


The teacher explains beforehand what is 
expected of the students. 


% 


o 2 


0 3 


o 4 


The teacher checks that students are 
concentrating while working on the reading 
assignment. 




q, 2 


0 3 


o 4 


The teacher discusses students’ work, after 
they have finished the reading assignment. 


Q* 


q 2 


0 3 


o 4 


The teacher tells students in advance how their 
work is going to be judged. 


o, 


o 2 


O, 


o 4 


The teacher asks whether every student has 
understood how to complete the reading 
assignment. 


c% 


o 2 


0 3 


o 4 


The teacher grades students’ work. 


O, 


o 2 


0 3 


o 4 


The teacher gives students the chance to ask 
questions about the reading assignment. 




o 2 


0 3 


o 4 


The teacher poses questions that motivate 
students to participate actively. 


0, 


q 2 


0 3 


o 4 


The teacher tells students how well they did 
on the reading assignment immediately after. 


ti 


q 2 


0 3 


o 4 
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SECTION 7: LIBRARIES 



In this section you are asked questions about libraries. These may be in your 
school and/or outside your school 

Q43 How often do you visit a library for the following activities? 

( Please darken only one circle in each row) 





Never 


A few 
times a 
year 


About 
once a 
month 


Several 
times a 
month 


Several 
times a 
week 


a) Borrow books to read for 
pleasure 


o, 


o 2 


o 3 


o 4 


o 5 


b) Borrow books for school work 


0. 


o 2 


0 3 


o 4 


o 5 


c) Work on homework, course 
assignments or research papers 


0, 


g 2 


o 3 


o 4 


G 5 


d) Read magazines or newspapers 


0, 


o 2 


o 3 


o; 4 


o 5 


e) Read books for fun 


o. 


o 2 


0 3 


g 4 


o 5 


f) Learn about things that are not 
course-related, such as sports, 
hobbies, people or music 


0, 


o 2 


o 3 


o 4 


0 5 


g) Use the Internet 


o. 


g 2 


o 3 


o 4 


o 5 



Q44 Does your school have a school library? 

No O, 

Yes Q 2 
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SECTION 8: YOUR STRATEGIES IN READING AND 



There are several approaches to studying and understanding texts. Some of them 
are more useful than others, depending on the kind of reading task. The next two 
questions present two reading tasks, followed by a list of these approaches or 
“ strategies . ” We want to know your opinion about the usefulness of these 
strategies for the different reading tasks. 

Both questions begin with a short description of a particular reading task. Then 
several possible reading strategies are listed. Think about the usefulness of each of 
the strategies in relation to the given reading task only. Some strategies may be 
useful for one reading task but not for another. 

Give a score between 1 and 6 to each strategy. A score of 1 means you think it is 
not a useful strategy at all for this reading task. A score of 6 means you think it is a 
very useful strategy for this reading task. 

You can use the same score more than once if you think two or more strategies are 
similarly useful, but please darken only one circle in each row. 



Here is an example question that a student has completed. ( This example is about 
playing table tennis, not reading.) 

Example Question 



Task: You want to improve at playing table tennis so you can win a local 
competition. 

How do you rate the usefulness of the following strategies for improving at playing 
table tennis? 



Possible strategy 



a) I read a book about table tennis 
technique. 

b) I practice playing table tennis against 
a friend as often as possible. 

c) I do general fitness exercises every 
morning. 

d) I watch expert players and try to 
figure out their techniques. 



Score 

Not useful 
at all 
( 1 ) 


(2) 


(3) 


(4) 


(5) 


Very 

usefiil 

( 6 ) 


Q 


• 


0 


o 


o 


O 


O 


o 


o 


0 


o 


• 


0 


o 


o 


• 


o 


o 


o 


o 


0 


• 


D 


0 
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Q45 Reading task: You have to understand and remember the 
information in a text. 

How do you rate the usefulness of the following strategies for 
understanding and memorizing the text? 



Possible strategy 



Score 

Not useful at 
all 



Very useful 





( 1 ) 


( 2 ) 


( 3 ) 


( 4 ) 


( 5 ) 


( 6 ) 


a) I concentrate on the parts of the 


o, 


o 2 


a* 


0 4 


Os 


06 


text that are easy to understand. 


b) I quickly read through the text 
twice. 


0, 


O, 


o 3 


0 4 


0 5 


06 


c) After reading the text, I discuss 
its content with other people. 


o, 


o 2 


Os 


0 4 


O 5 


O, 


d) I underline important parts of the 
text. 


o, 


Oj 


o 3 


0-4’ 


05 


( 0 S 


e) I summarize the text in my own 
words. 


o, 


o 2 


a* 


04 


05 


06 


f) I read the text aloud to another 
person. 


0, 


O* 


O 3 


O 4 


0 5 


o 6 
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Q46 Reading task: You have just read a long and rather difficult 

two-page text about fluctuations in the water level of a lake in 
Africa. You have to write a summary. 

How do you rate the usefulness of the following strategies for writing a 
summary of this two-page text? 



Possible strategy 



Score 

Not useful at 
all 



Very useful 





( 1 ) 


( 2 ) 


( 3 ) 


( 4 ) 


( 5 ) 


( 6 ) 


a) I write a summary. Then I check 
that each paragraph is covered in 
the summary, because the content 
of each paragraph should be 
included. 


o, 


o 2 


o 3 


0 4 


o 5 


0 6 


b) I try to copy out accurately as 
many sentences as possible. 


Oi 


o 2 


o 3 


o 4 


o 5 


0 6 


c) Before writing the summary, I 
read the text as many times as 
possible. 


O, 


o 2 


o 3 


o 4 


o 5 


o 6 


d) I carefully check whether the 
most important facts in the text 
are represented in the summary. 


O, 


o 2 


o 3 


o 4 


o 5 




e) I read through the text, 

underlining the most important 
sentences. Then I write them in 


o, 


o 2 


o 3 


o 4 


o 5 


o 6 



my own words as a summary. 



Thank you very much for your cooperation in 
completing this questionnaire! 
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ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT 



D.2 School Questionnaire 




OECD Program for 
International Student Assessment 2009 



OECD 
P I S A 



USA 

-Wain .Wv PISA .’OUV 



Date 



/ / 2009 



Month Day 



SCHOOL QUESTIONNAIRE 



Place Label Here 



SOM ID 
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ail be IcyM urafidedisl mi w il act he ill*. k>«ed m alemftAfc farm By law, noyoK »mi*a« oaUm NCFS amvc* » a*jed 
to a jail Km. a fine, u buck ifte or she «• i IEul> diaebaea ANY Mbnuitba thm muM idr*i/V yw 

Acdb4r| to ibe Fy n>iwi Rcdatna Ad tf I9M, b penui* m r ey iKd fe> mpoad to a ocllertinB ef ^owmw uaieaa lai 
oeledioR diaplaya a valal OMB «ml audn TVc valal 0MB cemiol main lea da Mr mi a lam rw>i,twti ■ IIS047S) 
The Ht maimed ki ctmadrtr ttia iaTaimKkn cedhxtne a raiiiadnrt to axtaac 30 wan pea icapome. aaelialim ibe taoe la 



t>: US DcpMUM nf Eicaim. WaabogMm, D C 20202 -465 1 If T« km c« 
w in* ia dm da .1 iab»irw mi thiz hrm. arm dire tK M : NtfairaJ Cola fa 

ef Eduutlbn 1900 K Swea. N W . Wabii^na. D C 20006-3630 

OMB Na IS3047S3. Apyausul Espbca 03/11/3011 



Core B Consortium 

Cito Institute for Educational Measurement 
University of Twente 

University of Jyvaskyla, Institute for Educational Research 
Direction de I' Evaluation de la Prospective et de la Performance (DEPP) 
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This questionnaire asks for information including: 

The structure and organization of the school; 

The student body and teachers; 

The school’s resources; 

The school’s instruction, curriculum and assessment; 

The school climate; 

The school policies and practices; 

The characteristics of the principal or designee. 

This information helps illustrate the similarities and differences between groups 
of schools in order to better establish the context for students’ test results. 

The questionnaire should be completed by the principal or designee. 

It should take about 30 minutes to complete. 

For some questions, specific expertise may be needed. You may consult experts 
to help you answer these questions. 

If you do not know an answer precisely, your best estimate will be adequate for 
the purposes of the study. 

Some questions ask about 10th grade or 10th graders. If you do not have a 
10th grade in your school, then answer these questions for the grade in 
your school that contains the most 15-year-olds. 

Your answers will be kept confidential. They will be combined with 
answers from other principals to calculate totals and averages in which no 
one school can be identified. 
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SECTION A: THE STRUCTURE AND ORGANIZATION OF THE 



Q1 Do you have the following grade levels in your school? 

( Please check one box in each row) 





Yes 


No 


a) Grade 1 


1 


□, 


b) Grade 2 


1 


2 


c) Grade 3 


1 


2 


d) Grade 4 


1 


2 


e) Grade 5 


1 


□, 


f) Grade 6 


1 


□, 


g) Grade 7 


1 


□, 


h) Grade 8 


1 


□, 


i) Grade 9 


1 


2 


j) Grade 10 


1 


□, 


k) Grade 11 


1 


□, 


1) Grade 12 


1 


□, 


m) Ungraded school 


— 1 1 


— 2 
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Q2 Is your school a public or a private school? 

(Please check only one box) 

A public school O ] 

A private school 2 

Q3 About what percentage of your total funding for a typical 
school year comes from the following sources? 

(Please write a number in each row. Write 0 ( zero) if no f unding comes 
from that source.) 

% 

a) Government (includes local, state and federal) 

b) Tuition, student fees, or school charges paid by parents 

c) Benefactors, donations, bequests, sponsorships, parent fundraising 

d) Other 

Total 100% 
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Q4 Which of the following definitions best describes the 
community in which your school is located? 

(Please check only one box ) 

A village, hamlet or rural area (fewer than 3,000 people) 

A small town (3,000 to about 15,000 people) l 2 

A town (15,000 to about 100,000 people) l 3 

A city (100,000 to about 1,000,000 people) l 4 

A large city (with over 1,000,000 people) l 5 



Q5a We are interested in the options parents have when choosing a 
school for their children. 

Which of the following statements best describes the 
schooling available to students in your location? 

(Please check only one box ) 

There are two or more other schools in this area that compete for our students. 
There is one other school in this area that competes for our students. 

There are no other schools in this area that compete for our students. 
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Q5b Other than your school, how many public and private schools 
in this area compete for your students? 

( Please write a number in each line. Write 0 (zero) if there are none.) 
a) Number of 

public schools: 



b) Number of 
private schools: 
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SECTION B: THE STUDENT BODY AND TEACHERS 



Q6 As of September 1 , 2009, what was the total school enrollment 
(number of students)? 

(Please write a number in each line. Write 0 (zero) if there are none.) 

a) Number of boys: 

b) Number of girls: 



Q7 About what percentage of students in your school repeated a 
grade, at these levels, last academic year? 

(Please write a number in each row. Write 0 (zero) if nobody repeated a 
grade. Check the ‘not available ’ box if the level does not exist in your 
school.) 



Level not 
available in 
% this school 



a) The approximate percentage of students repeating a grade 
at the middle/junior high school level (grades 7-9) in this 
school last year was: 

b) The approximate percentage of students repeating a grade 
at the high school level (grades 10-12) in this school last 
year was: 



996 



996 
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Q8 About how many students in the 10th grade in your 
school have a first language that is not English? 

( Please check only one box ) 

60% or more 

40% or more but less than 60% 

2 

20% or more but less than 40% I 

10% or more but less than 20% 

4 

More than 0%, but less than 10% l 5 

None 



Q9 How many of the following teachers are on the staff of your 
school? 

Include both full-time and part-time teachers. A full-time teacher is employed at least 90% of 
the time as a teacher for the full school year. All other teachers should be considered part-time. 



( Please write a number in each space provided. Write 0 (zero) if there are 
none.) 

Full-time Part-time 

a) Teachers in TOTAL 

b) Teachers fully certified by the state in the main assignment 
field 

c) Teachers with at least a bachelor’s degree 
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SECTION C: THE SCHOOL’S RESOURCES 

The goal of the following set of four questions is to gather information about the 
student -computer ratio for students in the 10th grade at your school. 



Number 



Q10a At your school, what is the total number of 
students in the 10th grade? 

QlOb Approximately, how many computers are 
available for these students for educational 
purposes? 

QIOc Approximately, how many of the computers 
listed in Question 10b are connected to the 
Internet/World Wide Web? 

QlOd Approximately, how many computers are 
available for all students in your school for 
educational purposes? 
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Q11 Is your school’s capacity to provide instruction hindered by 



any of the following issues? 

(Please check one box in each row) 


Not at 
all 


Very 

littfe 


To some 
extent 


A lot 


a) A lack of qualified science teachers 


1 


a 


3 


□. 


b) A lack of qualified mathematics teachers 


1 


2 


□, 


ft 


c) A lack of qualified English teachers 


1 


ft 


a* 




d) A lack of qualified teachers of other subjects 


1 


2 


ft 


ft 


e) A lack of library staff 


1 


a 


3 


4 


f) A lack of other support personnel 


1 


a 


3 


4 


g) Shortage or inadequacy of science laboratory 
equipment 


1 


a 


ft. 


ft 


h) Shortage or inadequacy of instructional 
materials (e.g., textbooks) 


1 


a 


3 


□. 


i) Shortage or inadequacy of computers for 
instruction 


1 


n 


3 


□. 


j) Lack or inadequacy of Internet connectivity 


1 


2 


Ok. 


ft 


k) Shortage or inadequacy of computer software 
for instruction 


1 


2 


□, 


ft 


1) Shortage or inadequacy of library materials 


1 


a 


3 


4 


m) Shortage or inadequacy of audio-visual 
resources 


1 


— 2 


A 


ft 
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SECTION D: SCHOOL CURRICULUM AND ASSESSMENT 



Q12 Some schools organize instruction differently for students with 
different abilities. What is your school’s policy about this for 
students in the 10th grade? 

(Please check one box in each row) 





For all 
subjects 


For some 
subjects 


Not for any 
subject 


a) Students are grouped by ability into different 
classes. 




z 


1 


2 


3 


b) Students are grouped by ability within their 
classes. 




z 


1 


□» 


□, 
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Q13 In this academic year, which of the following activities does 
your school offer to students in the 10th grade? 

( Please check one box in each row) 





Yes 


No 


a) Band, orchestra or choir 


1 




b) School play or school musical 


1 


Q 


c) School yearbook, newspaper or magazine 


1 


□: 


d) Volunteer or service activities 


1 


□: 


e) Book club 


1 




f) Debate club or debating activities 


1 




g) School club or school competition for 
foreign language, mathematics or science 


1 


□: 


h) Academic club (e.g., honor society) 


1 




i) Art club or art activities 


1 




j) Sports team or sports activities 


1 


□: 


k) Lectures and/or seminars (e.g., guest 
speakers such as writers or journalists) 


1 


□: 


1) Collaboration with local libraries 


1 




m) Collaboration with local newspapers 


1 


□: 


n) School-wide scheduled reading periods 


1 
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Q14 



Does your school offer any of the following options to students 
in the 10th grade whose first language is not English? 

(Please check one box in each row) 



Yes 

a) These students attend regular classes and 
receive additional periods of instruction 
aimed at developing English skills (e.g., 
reading literacy, grammar, vocabulary, 
communication). 

b) Before transferring to regular classes, 
these students attend a preparatory 
program aimed at developing English 
skills (e.g., reading literacy, grammar, 
vocabulary, communication). 

c) Before transferring to regular classes, 
these students receive some instruction in 

school subjects through their first 1 

language. 

d) These students receive significant amounts 

of instruction in their first language aimed — 

at developing proficiency in both 1 

languages. 

e) Class size is reduced to cater to the special — 

needs of these students. 1 



No 



2 






2 






2 



2 



2 
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Q15 Generally, in your school, how often are students in the 
10th grade assessed using the following methods? 



(Please check only one box in each row) 





Never 


1-2 
times a 
year 


3-5 
times a 
year 


Monthly 


More 
than 
once a 
month 


a) Standardized tests 


1 


a 


□, 


4 


r s 


b) Teacher-developed tests 


1 


2 


3 


a 




c) Teachers’ judgmental ratings 


1 


□. 


□, 


□ 4 


5 


d) Student portfolios 


1 


2 


3 


□ 4 




e) Student assignments/ 
projects/homework 


1 


□. 


□, 


4 


5 
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Q16 In your school, are assessments of students in the 10th 
grade used for any of the following purposes? 

( Please check only one box in each row) 



a) To inform parents about their child’s progress 

b) To make decisions about students’ retention or promotion 

c) To group students for instructional purposes 

d) To compare the school to district, state, or national 
performance 

e) To monitor the school’s progress from year to year 

f) To make judgments about teachers’ effectiveness 

g) To identify aspects of instruction or the curriculum 
that could be improved 

h) To compare the school with other schools 



Yes No 

□, ij 

1 2 



1 1 J 2 

, a 

1 2 

, a 

1 2 
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SECTION E: SCHOOL CLIMATE 

Q17 In your school, to what extent is the learning of students 
hindered by the following phenomena? 



(Please check one box in each row) 





Not at 
all 


Very 

little 


To some 
extent 


A 

lot 


a) Teachers’ low expectations of students 


1 


2 


□> 


a 


b) Student absenteeism 


1 


□. 


3 


□. 


c) Poor student-teacher relations 


1 


2 


3 


4 


d) Disruption of classes by students 


1 


2 


3 


4 


e) Teachers not meeting individual students’ needs 


1 


2 


3 


4 


f) Teacher absenteeism 


1 


2 






g) Students skipping classes 


1 


□. 


3 


□ 4 


h) Students lacking respect for teachers 


1 


2 


□3 


□ 4 


i) Staff resisting change 


1 


□. 


3 


r. 


j) Student use of alcohol or illegal drugs 


1 


2 


P» 


qj, 


k) Teachers being too strict with students 


1 


□. 


3 


4 


1) Students intimidating or bullying other students 


1 


2 


□3 


□ 4 


m) Students not being encouraged to achieve their full 
potential 


1 


□. 


3 


□ 4 
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Q18 Which statement below best characterizes parental 
expectations towards your school? 

(Please check only one box ) 

There is constant pressure from many parents, who expect our school to set 
very high academic standards and to have our students achieve them. 



Pressure on the school to achieve higher academic standards among students 
comes from a minority of parents. 






2 



Pressure from parents on the school to achieve higher academic standards 
among students is largely absent. 
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SECTION F: SCHOOL POLICIES AND PRACTICES 

Q19 How often are the following factors considered when students 
are admitted to your school? 

( Please check one box in each row) 





Never 


Sometimes 


Always 


a) Residence in a particular area 


1 


2 


□, 


b) Student’s record of academic performance 
(including placement tests) 


1 


2 


□, 


c) Recommendation of feeder schools 


1 


2 


□, 


d) Parents’ endorsement of the instructional 
or religious philosophy of the school 


1 


2 


□, 


e) Whether the student requires, or is 
interested in, a special program 


1 




n 


f) Preference given to family members of 
current or former students 


1 


2 


a, 


g) Other 


1 


- 2 


— 3 



2009 Data Files and Database with U.S. Specific Variables 



D-51 





Q20 In your school, how likely is it that a student in the 10th grade 
would be transferred to another school because of the 
following reasons? 

(Please check one box in each row) 

If students are never transferred, go to Q21. 





Not likely 


Likely 


Very likely 


a) Low academic achievement 


1 


□, 


3 


b) High academic achievement 


1 


2 


3 


c) Behavioral problems 


1 


2 


3 


d) Special learning needs 


1 


2 


3 


e) Parents’ or guardians’ request 


1 


□, 


□. 


f) Other 


1 


— 2 


— 3 
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Q21 This set of questions explores aspects of the school’s 
accountability to parents. 

(Please check one box in each row) 



Yes 

a) Does your school provide information to parents of students in the 
10th grade on their child’s academic performance relative to other 
students in the 10th grade in your school? 

b) Does your school provide information to parents of students in the 
10th grade on their child’s academic performance relative to national 
or state benchmarks? 



c) Does your school provide information to parents on the academic 
performance of students in the 10th grade as a group relative to 
students in the same grade in other schools? 
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Q22 



In your school, are achievement data used in any of the 
following ways? 

Achievement data include aggregated school or grade-level test scores or 
grades, or graduation rates. 

(Please check one box in each row) 



a) Achievement data are posted publicly (e.g., in the media). 

b) Achievement data are used in evaluation of the principal’s 
performance. 

c) Achievement data are used in evaluation of teachers’ performances. 

d) Achievement data are used in decisions about instructional resource 
allocation to the school. 



Yes No 

1 2 

a, □, 
fe □, 

— 1 1 2 



e) Achievement data are tracked over time by an administrative authority 
(such as a district, state, or federal education agency). 
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Q23 During the last year, have any of the following methods 
been used to monitor the practice of English teachers at 
your school? 

( Please check one box in each row) 



Yes No 



a) Tests or assessments of student 
achievement 



b) Teacher peer review (of lesson plans, 
assessment instruments, lessons) 



1 






2 



c) Principal or senior staff observations of 
lessons 

d) Observation of classes by other persons 
external to the school 
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Q24 Regarding your school, who has a considerable responsibility 
for the following tasks? 



( Please check as many boxes as appropriate in each row) 



School-level State or local 
governing education 
Principals Teachers board agency 



a) Selecting teachers for hire 

b) Firing teachers 

c) Establishing teachers’ 
starting salaries 

d) Determining teachers’ 
salary increases 

e) Formulating the school 
budget 

f) Deciding on budget 
allocations within the 
school 

g) Establishing student 
disciplinary policies 

h) Establishing student 
assessment policies 

i) Approving students for 
admission to the school 

j) Choosing which textbooks 
are used 

k) Determining course 
content 



U.S. 

Department 
of Education 



1 



1 



1 



i 



i 



i 



i 



i 



i 



i 
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Q25 



1) Deciding which courses are 
offered 



Regarding your school, which of the following bodies exert a 
direct influence on decision making about staffing, budgeting, 
instructional content and assessment practices? 

( Please check as many boxes as apply) 



Area of influence 



Staffing 



Budgeting 



Instructional 

content 



Assessment 

practices 



a) State or federal education 
agencies (e.g., state education 
department) 

b) Local education agency or 
local school board 

c) School-level governing board 

d) Parent groups 

e) Teacher groups 
(e.g., staff association, 
curriculum committees, trade 
union) 

f) Student groups 

(e.g., student association, 
youth organization) 

g) External examination 
boards 
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Q26 



a) 



b) 



c) 

d) 



e) 



f) 

g) 



h) 



Below you can find statements about your management of this 
school. Please indicate the frequency of the following activities 
and behaviors in your school during the last school year . 

(Please check only one box in each row) 



Never Seldom Quite often Very often 



I make sure that the professional 
development activities of teachers 
are in accordance with the teaching 
goals of the school. 



I ensure that teachers work 
according to the school’s 
educational goals. 

I observe instruction in classrooms. 



I use student performance results to 
develop the school’s educational 
goals. 

I give teachers suggestions as to 
how they can improve their 
teaching. 



I monitor students’ work. 



When a teacher has problems in 
his/her classroom, I take the 
initiative to discuss matters. 

I inform teachers about 
possibilities for updating their 
knowledge and skills. 



— 2 4 

n 2 □, n. 



□, 








n 2 c 


l 


=■ 


n 2 c 




=■ 


n 2 : 


1 


n 
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i) I check to see whether classroom 


Never 


Seldom 


Quite often 


Very often 


activities are in keeping with our 
educational goals. 

j) I take exam results into account in 


1 


a 


3 


d. 


decisions regarding curriculum 
development. 

k) I ensure that there is clarity 


1 


□, 


0. 


4 


concerning the responsibility for 
coordinating the curriculum. 

1) When a teacher brings up a 


1 




d, 


4 


classroom problem, we solve the 
problem together. 

m) I pay attention to disruptive 


1 




□ □ □ 


4 


behavior in classrooms, 
n) I take over lessons from teachers 


1 


2 


4 


who are unexpectedly absent. 


1 


2 


4 



Q27 Approximately what percentage of students at this 

school last year were eligible for free- or reduced-price 
lunches through the National School Lunch Program? 

(Please write a number on the line. Write 0 (zero) if there are 
none.) 

Percentage of students eligible 
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SECTION G: ABOUT YOU 



Q28 Are you female or male? 

Female Male 



1 1 2 



Thank you very much for your cooperation in 
completing this questionnaire! 
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Appendix E. 
Data Collection Training Agenda 
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Appendix E. Data Collection Training Agenda 




PISA 

USA 



Program for International Student Assessment 

Test Administrator Training Agenda 
September 11-12, 2009 



DAY 

Friday, September 11 



TIME 


TOPIC 


9:00 - 10:00 


SESSION 1 

Introduction of attendees 
Introduction to PISA 


10:00 - 10:45 


SESSION 2 

PISA materials and supplies 
Security and Confidentiality 


10:45 - 11:00 


SHORT BREAK 


11:00 - 12:00 


SESSION 3 

Sampling and Tracking Forms 


12:00 - 1:00 


LUNCH BREAK 

ID Pictures and laptop set up 


1:00 - 2:00 


SESSION 4 

Pre-Assessment Call 


2:00 - 3:00 


SESSION 5 

Introduction to FMS 


3:00 - 3:20 


SHORT BREAK 


3:20 - 5:00 


SESSION 6 



Student Questionnaire preparation 

Arriving at the school/Meet with SC/Set up room 

Test booklet preparation 

5:00-5:20 DAY 1 WRAP UP 

Shuttle arrives at Westat at 5:30 
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Appendix E. Data Collection Training Agenda 



Saturday, September 12 

9:00 - 9:30 


REVIEW and Q/A 


9:30 - 10:30 


SESSION 7 

Conducting the PISA Assessment 
Student Questionnaire QxQ 


10:30 - 10:40 


SESSION 8 

Session ID Codes 


10:40 - 11:00 


SHORT BREAK 


11:00 - 11:30 


SESSION 9 

Ending the session 

Concluding the session (for now) 

"Participation rates" & Make-up sessions 


11:30 - 12:00 


SESSION 10 

Packing the session box 


12:00 - 12:45 


LUNCH BREAK 


12:45 - 1:20 


SESSION 11 

Post-Assessment FMS update 


1:20 - 1:40 


SESSION 12 

AA Training 


1:40 - 2:00 


SESSION 13 

Administrative Duties 


2:00 - 2:15 


SHORT BREAK 


2:15 - 2:30 


MEET WITH FM 


2:30 - 2:50 


DAY 2 WRAP UP 

Shuttle arrives at Westat at 3:00 
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Appendix F. Item Reliabilities 



Table F.l. Item Reliabilities 



Item name 


Booklet * 


Number multiple 
scored 


Reliability (percent] 


Cluster number R1 


R219Q01A 


2 


163 


100.0 


R219Q01B 


2 


163 


100.0 


R219Q01C 


2 


163 


100.0 


R219Q01D 


2 


163 


100.0 


R219Q01E 


2 


163 


99.4 


R219Q02 


2 


163 


99.4 


R067Q04 


2 


163 


97.9 


R067Q05 


2 


163 


98.2 


R102Q04A 


2 


163 


100.0 


R102Q05 


2 


163 


99.7 


R220Q01 


2 


163 


99.4 


Cluster number R2 


R227Q03 


8 


162 


99.7 


R227Q06 


8 


162 


100.0 


R111Q02B 


8 


162 


96.6 


R111Q06B 


8 


162 


98.5 


R055Q02 


8 


162 


99.4 


R055Q03 


8 


162 


100.0 


R055Q05 


8 


162 


99.1 


R104Q01 


8 


162 


100.0 


R104Q02 


8 


162 


99.7 


R104Q05 


8 


162 


99.4 


Cluster number R3 


R458Q07 


4 


165 


99.7 


R447Q06 


4 


165 


99.4 


R452Q03 


4 


165 


100.0 


R452Q06 


4 


165 


100.0 


R414Q06 


4 


165 


100.0 


Cluster number R4 


R083Q02 


5 


164 


100.0 


R083Q03 


5 


164 


100.0 


R442Q02 


5 


164 


99.7 


R442Q03 


5 


164 


99.7 


R442Q05 


5 


164 


97.6 


R442Q06 


5 


164 


98.5 


R245Q01 


5 


164 


100.0 


R245Q02 


5 


164 


100.0 


Cluster number R5 


R404Q10A 


6 


162 


100.0 


R404Q10B 


6 


162 


99.4 


R406Q01 


6 


162 


99.4 


R406Q05 


6 


162 


99.4 


R406Q02 


6 


162 


99.4 
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Appendix F. Item Reliabilities 



Table F.l. Item Reliabilities 



Item name 


Booklet * 


Number multiple 
scored 


Reliability (percent] 


R455Q02 


6 


162 


99.1 


R455Q03 


6 


162 


100.0 


Cluster number R6 


R420Q02 


7 


164 


99.7 


R420Q10 


7 


164 


99.1 


R420Q06 


7 


164 


99.1 


R420Q09 


7 


164 


100.0 


R453Q04 


7 


164 


100.0 


R453Q06 


7 


164 


100.0 


R412Q08 


7 


164 


99.7 


R437Q07 


7 


164 


99.4 


Cluster number R7 


R456Q02 


12 


162 


99.4 


R456Q06 


12 


162 


100.0 


R466Q02 


12 


162 


99.7 


R466Q06 


12 


162 


100.0 


R446Q03 


12 


162 


100.0 


R446Q06 


12 


162 


100.0 


R432Q01 


12 


162 


100.0 


R432Q05 


12 


162 


100.0 


R460Q01 


12 


162 


100.0 


Cluster number Ml 


M474Q01 


1 


166 


100.0 


M 155Q02T ens 


1 


166 


93.1 


M 155Q02O nes 


1 


166 


92.8 


M155Q01 


1 


166 


93.4 


M 155Q03T ens 


1 


166 


93.4 


M 155Q03O nes 


1 


166 


92.2 


M411Q01 


1 


166 


99.7 


M442Q02 


1 


166 


991 


M 462Q01T ens 


1 


166 


97.0 


M 462Q01O nes 


1 


166 


94.6 


Cluster number M2 


M446Q01 


9 


165 


99.4 


M446Q02 


9 


165 


99.4 


M828Q01 


9 


165 


93.0 


M828Q02 


9 


165 


99.4 


M828Q03 


9 


165 


98.8 


Cluster number M3 


M496Q02 


11 


162 


98.8 


M406Q01 


11 


162 


97.5 


M406Q02 


11 


162 


97.8 
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Appendix F. Item Reliabilities 



Table F.l. Item Reliabilities 



Item name 


Booklet * 


Number multiple 
scored 


Reliability (percent] 


Cluster number SI 


S465Q01 


3 


166 


95.5 


S131Q02Tens 


3 


166 


98.5 


S131Q02Ones 


3 


166 


95.5 


S131Q04Tens 


3 


166 


98.8 


S131Q04Ones 


3 


166 


96.1 


S428Q05 


3 


166 


95.8 


S514Q02 


3 


166 


98.8 


S514Q03 


3 


166 


97.3 


S514Q04 


3 


166 


99.1 


S438Q03Tens 


3 


166 


94.0 


S438Q03Ones 


3 


166 


92.8 


Cluster number S2 


S269Q01 


10 


165 


98.5 


S269Q03Tens 


10 


165 


99.1 


S269Q03O nes 


10 


165 


98.2 


S408Q03 


10 


165 


98.8 


S519Q01 


10 


165 


94.8 


S519Q03 


10 


165 


95.8 


Cluster number S3 


S326Q01 


13 


163 


96.6 


S326Q02 


13 


163 


98.2 


S413Q06 


13 


163 


100.0 


S498Q04 


13 


163 


98.2 


S425Q03 


13 


163 


96.3 


S425Q04 


13 


163 


99.7 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for International StudentAssessment 
(PISA), 2009. 
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Appendix G. 
Explanation of Indices 
(abstracted from OECD 2010c and 

OECD 2010d) 
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Appendix G. Explanation of Indices (abstracted from OECD 2010c and OECD 2010d) 



This section explains the indices derived from the student, school and parent context questionnaires used in PISA 2009, Parentquestionnaire indices are only 
available forthe 14 countries that chose to administerthe optional parentquestionnaire. 

Several PISA measures reflect indices that summarize responses from students, their parents or school representatives (typically principals) to a series of related 
questions. The questions were selected from a largerpool of questions on the basis of theoretical considerations and previous research. Structural equation 
modeling was used to confirm the theoretically expected behavior of the indices and to validate their comparability across countries. For this purpose, a model 
was estimated separately foreach country and collectively for all OECD countries. 

Fora detailed description of other PISA indices and details on the methods, see PISA 2009 Technical Report (OECD, forthcoming). 

There are two types of indices: simple indices and scale indices. 

Simple indices are the variables that are constructed through the arithmetic transformation or recoding of one or more items, in exactly the same way across 
assessments. Flere, item responses are used to calculate meaningful variables, such as the recoding of the four-digit ISCO-88 codes into "FI ighest parents’ 
socio-economic index (FI IS E I)" or, teacher-student ratio based on information from the school questionnaire. 

Scale indices are the variables constructed through the scaling of multiple items. Unless otherwise indicated, the index was scaled using a weighted maximum 
likelihood estimate (WLE) (Warm, 1985), using a one-parameter item response model (a partial credit model was used in the case of items with more than two 
categories). 

The scaling was done in three stages: 

•The item parameters were estimated from equal-sized subsamples of students from each OECD country. 

•The estimates were computed for all students and all schools by anchoring the item parameters obtained in the preceding step. 

•The indices were then standardized so that the mean of the index value forthe OECD student population was zero and the standard deviation was one 
(countries being given equal weight in the standardization process). 

Sequential codes were assigned to the different response categories of the questions in the sequence in which the latter appeared in the student, school or 
parent questionnaires. Where indicated in this section, these codes were inverted forthe purpose of constructing indices or scales. It is important to note that 
negative values for an index do not necessarily imply that students responded negatively to the underlying questions, A negative value merely indicates that the 
respondents answered less positively than all respondents did on average across OECD countries. Likewise, a positive value on an index indicates thatthe 
respondents answered more favorably, or more positively, than respondents did, on average, in OECD countries. Terms enclosed in brackets <>in the following 
descriptions were replaced in the national versions of the student, school and parent questionnaires by the appropriate national equivalent. For example, the term 
qualification atlSCED level 5A>was translated in the United States into "Bachelor's degree, post-graduate certificate program, Master's degree program orfirst 
professional degree program". Similarly the term <classes in the language of assessment* in Luxembourg was translated into "German classes" or "French 
classes" depending on whether students received the German or French version of the assessment instruments. 

In addition to simple and scaled indices described in this annex, there are a number of variables from the questionnaires that correspond to single items notused 
to construct indices. These non-recoded variables have prefix of "ST"forthe questionnaire items in the student questionnaire, "SC "for the items in the school 
questionnaire, and "PA” for the items in the parentquestionnaire. All the context questionnaires as well as the PISA international database, including all 
variables, are available through www.pisa.oecd.org . 

Student-level simple indices 

Age 

The variable AGE is calculated as the difference between the middle month and the year in which students were assessed and their month and year of birth, 
expressed in years and months. 

Study program 

In PISA 2009, study programs available to 15-year-old students in each country were collected both through the student tracking form and the student 
questionnaire (ST02). All study programs were classified using ISCED (OECD, 1999). In the PISA international database, all national programs are indicated in a 
variable (PROGN) where the first three digits are the ISO code fora country, the fourth digit the sub-national category and the last two digits the nationally 
specific program code. 

The following internationally comparable indices were derived from the data on study programs: 

•Program level (ISCEDL) indicates whether students are (1) primary education level (ISCED 1); (2) lower secondary education level; or (3) upper secondary 
education level. 

•Program designation (ISCEDD) indicates the designation of the study program: (1) = "A" (general programs designed to give access to the next program level); 
(2) ="B" (programs designed to give access to vocational studies at the next program level); (3) = "C " (programs designed to give direct access to the labor 
market); or (4) ="M" (modular programs that combine any or all of these characteristics). 

•Program orientation (ISCEDO) indicates whether the program's curricular content is (1) general; (2) pre-vocational; (3) vocational; or (4) modular programs that 
combine any orall of these characteristics. 
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Appendix G. Explanation of Indices (abstracted from OECD 2010c and OECD 2010cl) 



Occupational status of parents 

Occupational data for both a student's father and a student's mother were obtained by asking constructed-response questions in the student questionnaire 
(ST9a, ST9b, ST12, ST13a, ST13b and ST16). The responses were coded to four-digit ISCO codes (ILO, 1990) and then mapped to Ganzeboom, etal.'s (1992) 
SEI index. Higher scores of SEI indicate higher levels of occupational status. The following three indices are obtained: 

• Mother's occupational status (BMMJ ). 

• Father's occupational status (BFMJ ). 

•The highest occupational level of parents (HISEI) corresponds to the higherSEI score of either parent or to the only available parent's SEI score. 

Educational level of parents 

The educational level of parents is classified using ISCED (OECD, 1999) based on students' responses in the student questionnaire (ST10, ST11, STM and 
ST15). Please note that the question format for school education in PISA 2009 differs from the one used in PISA 2000, 2003 and 2006 but the method used to 
compute parental education is the same. 

As in PISA 2000, 2003 and 2006, indices were constructed by selecting the highest level for each pa rent and then assigning them to the following categories: (0) 
None, (1) ISCED 1 (primary education), (2) ISCED 2 (lower secondary), (3) ISCED Level 3B or 3C (vocationa l/pre-voca tiona I upper secondary), (4) ISCED 3A 
(upper secondary) and/or ISCED 4 (non-tertiary post-secondary), (5) ISCED 5B (vocational tertiary), (6) ISCED 5A, 6 (theoretically oriented tertiary and post- 
graduate). The following three indices with these categories are developed: 

• Mother's educational level (MISCED). 

•Father's educational level (FISCED). 

• Highest educational level of parents (H ISCED) corresponds to the higher ISCED level of either parent. 

Highest educational level of parents was also converted into the numberof years of schooling (PARED). For the conversion of level of education into years of 
schooling, see Table Al.l. 

Immigration and language background 

Information on the country of birth of students and their parents (ST17) is collected in a similar manner as in PISA 2000, PISA 2003 and PISA 2006 by using 
nationally specific ISO coded variables. The ISO codes of the country of birth for students and their parents are available in the PISA international database 
(COBN S, COBN M, and COBN F). 

The index on immigrant background (IM MIG) has the following categories: (1) native students (those students born in the country of assessment, or those with at 
least one parent born in that country: students who were born abroad with at least one parent born in the country of assessment are also classified as 'native' 
students), (2) second-generation students (those born in the country of assessment but whose parents were born in another country) and (3) first-generation 
students (those born outside the country of assessment and whose parents were also born in another country). Students with missing responses for either the 
student or for both parents, or for all three questions have been given missing values for this variable. 

Students indicate the language they usually speak at home. The data are captured in nationally-specific language codes, which were recorded into variable 
ST19Q01 with the following two values: (1) language athome is the same as the language ofassessmentand (2) language athome is a different language than 
the language ofassessment 
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Appendix G. Explanation of Indices (abstracted from OECD 2010c and OECD 2010d) 



Table Al.l. Levels of parental education converted into years of schooling 





Did not go 
to school 


Completed 
ISCED Level 1 
(primary 
education) 


Completed ISCED 
Level 2 (lower 
secondary 
education) 


Completed ISCED Levels3B 
or3C (upper secondary 
education providing direct 
access to the labor market or 
to ISCED 5B programmes) 


Completed ISCED Level 3A 
(upper secondary education 
providing access to ISCED 5A 
and 5B programmes) and/or 
ISCED Level 4 (nontertiary 
post-secondary) 


Completed ISCED Level 
5A (university level 
tertiary education) or 
ISCED Level 6 (advanced 
research programmes) 


Completed ISCED 
Level 5B (non- 
university tertiary 
education) 


q Australia 


0.0 


6.0 


10.0 


11.0 


12.0 


15.0 


14.0 


jj Austria 


0.0 


4.0 


9.0 


12.0 


12.5 


17.0 


15.0 


° Belgium 


0.0 


6.0 


9.0 


12.0 


12.0 


17.0 


14.5 


Canada 


0.0 


6.0 


9.0 


12.0 


12.0 


17.0 


15.0 


Chile 


0.0 


6.0 


8.0 


12.0 


12.0 


17.0 


16.0 


Czech Republic 


0.0 


5.0 


9.0 


11.0 


13.0 


16.0 


16.0 


Denmark 


0.0 


6.0 


9.0 


12.0 


12.0 


17.0 


15.0 


Estonia 


0.0 


4.0 


9.0 


12.0 


12.0 


16.0 


15.0 


Finland 


0.0 


6.0 


9.0 


12.0 


12.0 


16.5 


14.5 


France 


0.0 


5.0 


9.0 


12.0 


12.0 


15.0 


14.0 


Germany 


0.0 


4.0 


10.0 


13.0 


13.0 


18.0 


15.0 


Greece 


0.0 


6.0 


9.0 


11.5 


12.0 


17.0 


15.0 


Hungary 


0.0 


4.0 


8.0 


10.5 


12.0 


16.5 


13.5 


Iceland 


0.0 


7.0 


10.0 


13.0 


14.0 


18.0 


16.0 


Ireland 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


14.0 


Israel 


0.0 


6.0 


9.0 


12.0 


12.0 


15.0 


15.0 


Italy 


0.0 


5.0 


8.0 


12.0 


13.0 


17.0 


16.0 


Japan 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


14.0 


Korea 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


14.0 


Luxembourg 


0.0 


6.0 


9.0 


12.0 


13.0 


17.0 


16.0 


Mexico 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


14.0 


Netherlands 


0.0 


6.0 


10.0 


a 


12.0 


16.0 


a 


New Zealand 


0.0 


5.5 


10.0 


11.0 


12.0 


15.0 


14.0 


Norway 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


14.0 


Poland 


0.0 


a 


8.0 


11.0 


12.0 


16.0 


15.0 


Portugal 


0.0 


6.0 


9.0 


12.0 


12.0 


17.0 


15.0 


Scotland 


0.0 


7.0 


11.0 


13.0 


13.0 


16.0 


16.0 


Slovak Republic 


0.0 


4.5 


8.5 


12.0 


12.0 


17.5 


13.5 


Slovenia 


0.0 


4.0 


8.0 


11.0 


12.0 


16.0 


15.0 


Spain 


0.0 


5.0 


8.0 


10.0 


12.0 


16.5 


13.0 


Sweden 


0.0 


6.0 


9.0 


11.5 


12.0 


15.5 


14.0 


Switzerland 


0.0 


6.0 


9.0 


12.5 


12.5 


17.5 


14.5 


Turkey 


0.0 


5.0 


8.0 


11.0 


11.0 


15.0 


13.0 


United Kingdom 


0.0 


6.0 


9.0 


12.0 


13.0 


16.0 


15.0 


United States 


0.0 


6.0 


9.0 


a 


12.0 


16.0 


14.0 



Albania 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


16.0 


Argentina 


0.0 


6.0 


10.0 


12.0 


12.0 


17.0 


14.5 


Azerbaijan 


0.0 


4.0 


9.0 


11.0 


11.0 


17.0 


14.0 


Brazil 


0.0 


4.0 


8.0 


11.0 


11.0 


16.0 


14.5 


Bulgaria 


0.0 


4.0 


8.0 


12.0 


12.0 


17.5 


15.0 


Colombia 


0.0 


5.0 


9.0 


11.0 


11.0 


15.5 


14.0 


Croatia 


0.0 


4.0 


8.0 


11.0 


12.0 


17.0 


15.0 


Dubai (UAE) 


0.0 


5.0 


9.0 


12.0 


12.0 


16.0 


15.0 


Hong Kong-China 


0.0 


6.0 


9.0 


11.0 


13.0 


16.0 


14.0 


Indonesia 


0.0 


6.0 


9.0 


12.0 


12.0 


15.0 


14.0 


1 ordan 


0.0 


6.0 


10.0 


12.0 


12.0 


16.0 


14.5 


Kazakhstan 


0.0 


4.0 


9.0 


11.5 


12.5 


15.0 


14.0 


Kyrgyzstan 


0.0 


4.0 


8.0 


11.0 


10.0 


15.0 


13.0 


Latvia 


0.0 


3.0 


8.0 


11.0 


11.0 


16.0 


16.0 


Liechtenstein 


0.0 


5.0 


9.0 


11.0 


13.0 


17.0 


14.0 


Lithuania 


0.0 


3.0 


8.0 


11.0 


11.0 


16.0 


15.0 


Macao-China 


0.0 


6.0 


9.0 


11.0 


12.0 


16.0 


15.0 


Montenegro 


0.0 


4.0 


8.0 


11.0 


12.0 


16.0 


15.0 


Panama 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


a 


Peru 


0.0 


6.0 


9.0 


11.0 


11.0 


17.0 


14.0 


Qatar 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


15.0 


Romania 


0.0 


4.0 


8.0 


11.5 


12.5 


16.0 


14.0 


Russian Federation 


0.0 


4.0 


9.0 


11.5 


12.0 


15.0 


a 


Serbia 


0.0 


4.0 


8.0 


11.0 


12.0 


17.0 


14.5 


Shanghai-China 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


15.0 


Singapore 


0.0 


6.0 


8.0 


10.5 


10.5 


12.5 


12.5 


Chinese Taipei 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


14.0 


Thailand 


0.0 


6.0 


9.0 


12.0 


12.0 


16.0 


14.0 


Trinidad and Tobago 


0.0 


5.0 


9.0 


12.0 


12.0 


16.0 


15.0 


Tunisia 


0.0 


6.0 


9.0 


12.0 


13.0 


17.0 


16.0 


Uruguay 


0.0 


6.0 


9.0 


12.0 


12.0 


17.0 


15.0 



StatLink http://dx.doi.org/10.1787/888932343171 
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Appendix G. Explanation of Indices (abstracted from OECD 2010c and OECD 2010cl) 



Family structure 

The index of family structure (FAMSTRUC) is based on students' responses regarding people living at home with them (ST08). This index has the following three 
values: (1) single-parentfamily (students living with only one of the following: mother, father, male guardian, female guardian), (2) two-pa rentfamily (students 
living with a father or step/foster father and a mother or step/foster mother) and (3) other (except the non-responses, which are coded as missing ornot 
applicable). 

Relative grade 

Data on the student's grade are obtained both from the student questionnaire (ST01) and from the student tracking form. As with all variables thatare on both the 
tracking form and the questionnaire, inconsistencies between the two sources are reviewed and resolved during data-cleaning. In order to capture between- 
country variation, the relative grade index (GRADE) indicates whether students are at the modal grade in a country (value of 0), or whether they are below or 
above the modal grade level (+ x grades, - x grades). 

The relationship between the grade and student performance was estimated through a multilevel model accounting for the following background variables: i) the 

PISA index of economic, social and cultural status; ii) the PISA index of economic, social and cultural status squared; iii) the school mean of the PISA 
index of economic, social and cultural status; iv) an indicator as to whether students were foreign born first-generation students; v) the percentage of first- 
generation students in the school; and vi) students' gender. 

Table A1.2 presents the results of the multilevel model. Column 1 in Table A1.2 estimates the score pointdifference that is associated with one grade level (or 
school year). This difference can be estimated for the 32 OECD countries in which a sizeable number of 15-year-olds in the PISA samples were enrolled in at 
least two different grades. Since 15-year-olds cannotbe assumed to be distributed atrandom across the grade levels, adjustments had to be made forthe 
above-mentioned contextual factors that may relate to the assignment of students to the different grade levels. These adjustments are documented in columns 2 
to 7 of the table. While it is possible to estimate the typical performance difference among students in two adjacent grades net of the effects of selection and 
contextual factors, this difference cannot automatically be equated with the progress that students have made over the last school year but should be interpreted 
as a lowerboundary of the progress achieved. This is not only because different students were assessed but also because the content of the PISA assessment 
was not expressly designed to match what students had learned in the preceding school yearbutmore broadly to assess the cumulative outcome of learning in 
school up to age 15. For example, if the curriculum of the grades in which 15-year-olds are enrolled mainly includes material other than that assessed by PISA 
(which, in turn, may have been included in earlier school years) then the observed performance difference will underestimate student progress. 

Learning time 

Learning time in testlanguage (LMINS) was computed by multiplying students' responses on the number of minutes on average in the testlanguage class by 
number of test language class periods per week (ST28 and ST29). Comparable indices are computed for mathematics (MM INS) and science (SMINS). 

Student-level scale indices 

Family wealth 

The index of family wealth (WEALTH ) is based on the students' responses on whether they had the following at home: a room of their own, a link to the Internet, 
a dishwasher (treated as a country-specific item), a DVD player, and three other country-specific items (some items in ST20); and their responses on the number 
of cellular phones, televisions, computers, cars and the rooms with a bath or shower (ST21). 

Home educational resources 

The index of home educational resources (HEDRES) is based on the items measuring the existence of educational resources at home including a desk and a 
quietplace to study, a computer that students can use forschoolwork, educational software, books to help with students' school work, technical reference books 
and a dictionary (some items in ST20). 

Cultural possessions 

The index of cultural possessions (CULT POSS) is based on the students' responses to whether they had the following at home: 
classic literature, books of poetry and works of art (some items in ST20). 
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Appendix G. Explanation of Indices (abstracted from OECD 2010c and OECD 2010d) 



Table A1.2. A multilevel model to estimate grade effects in reading, accounting for some background variables 





Grade 


Index of 

economic, social 
and cultural 
status 


Index of 

economic, social 
and cultural 
status squared 


School 
mean index 
of economic, 
social and 
cultural status 


First Generation 
students 


School percentage 
of first generation 
students 


Gender - student 
is a girl 


Intercept 


Coef. 


S.E. 


Coef. 


S.E. 


Coef. 


S.E. 


Coef. 


S.E. 


Coef. 


S.E. 


Coef. 


S.E. 


Coef. 


S.E. 


Coef. 


S.E. 


Q Australia 


33.2 


(1.95) 


30.0 


(1.36) 


-3.8 


(1.05) 


66.4 


(1.87) 


-7.4 


(2.82) 


0.1 


(0.07) 


32.9 


(1.91) 


466.0 


(1.39) 


ju Austria 


35.3 


(2.18) 


11.4 


(1.66) 


0.5 


(1.00) 


89.7 


(3.86) 


-33.1 


(6.11) 


1.4 


(0.13) 


19.9 


(2.67) 


467.9 


(2.45) 


0 Belgium 


48.9 


(1.98) 


10.0 


(1.12) 


-0.1 


(0.63) 


79.9 


(1.73) 


-3.2 


(5.18) 


0.3 


(0.11) 


11.3 


(1.81) 


507.0 


(1.70) 


Canada 


45.0 


(2.14) 


19.4 


(1.52) 


1.5 


(0.91) 


33.9 


(2.28) 


-13.7 


03.18 


0.3 


(0.04) 


30.4 


(1.60) 


483.4 


(1.76) 


Chile 


35.5 


(1.55) 


8.6 


(1.52) 


0.3 


(0.63) 


37.4 


(1.61) 


c 


c 


c 


C 


13.8 


(2.33) 


478.6 


(1.60) 


Czech Republic 


44.6 


(3.39) 


13.4 


(1.89) 


-2.3 


(1.47) 


111.5 


(3.12) 


-8.9 


(12.29) 


0.4 


(0.33) 


32.3 


(2.84) 


460.7 


(2.39) 


Denmark 


36.1 


(3.02) 


27.9 


(1.51) 


-2.8 


(1.10) 


35.1 


(2.91) 


-37.5 


(5.97) 


0.0 


(0.14) 


25.5 


(2.59) 


474.0 


(1.95) 


Estonia 


44.4 


(2.74) 


14.1 


(1.80) 


1.6 


(1.43) 


52.1 


(4.52) 


-18.7 


(14.08) 


-3.3 


(0.44) 


36.7 


(2.45) 


485.8 


(2.02) 


Finland 


37.3 


(3.60) 


27.7 


(1.66) 


-2.5 


(1.30) 


10.4 


(3.28) 


-56.0 


(13.09) 


-0.1 


(0.29) 


51.5 


(2.26) 


500.6 


(2.02) 


France 


47.1 


(5.14) 


12.5 


(1.70) 


-1.9 


(1.12) 


81.6 


(4.04) 


-11.6 


(9.24) 


0.2 


(0.15) 


25.9 


(2.67) 


516.5 


(2.35) 


Germany 


34.4 


(1.74) 


9.2 


(1.23) 


-1.6 


(0.74) 


109.1 


(2.16) 


-13.2 


(4.80) 


0.2 


(0.12) 


27.2 


(1.92) 


458.0 


(1.46) 


Greece 


22.6 


(10.86) 


15.9 


(1.46) 


1.5 


(1.07) 


41.2 


(2.84) 


-15.0 


(7.82) 


0.0 


(0.18) 


36.2 


(2.55) 


469.0 


(2.04) 


Hungary 


25.6 


(2.19) 


8.3 


(1.39) 


0.9 


(0.87) 


74.8 


(2.09) 


2.8 


(7.92) 


0.0 


(0.27) 


21.4 


(2.22) 


494.1 


(1.65) 


Iceland 


c 


c 


29.8 


(2.56) 


-5.1 


(1.56) 


-3.8 


(5.12) 


-52.2 


(11.45) 


-1.3 


(0.40) 


44.9 


(2.59) 


469.1 


(4.23) 


Ireland 


18.2 


(1.99) 


29.7 


(1.78) 


-3.5 


(1.44) 


43.6 


(2.68) 


-32.8 


(6.52) 


-0.1 


(0.20) 


33.9 


(3.62) 


474.8 


(2.77) 


Israel 


36.6 


(3.85) 


19.9 


(1.90) 


3.4 


(1.04) 


104.7 


(2.10) 


-11.0 


(6.13) 


1.5 


(0.08) 


29.4 


(2.81) 


460.1 


(2.13) 


Italy 


36.1 


(1.67) 


4.5 


(0.69) 


-1.4 


(0.42) 


76.4 


(1.07) 


-29.7 


(3.36) 


0.2 


(0.08) 


24.0 


(1.29) 


491.4 


(0.85) 


Japan 


a 


a 


4.1 


(1.51) 


0.1 


(1.47) 


144.2 


(2.40) 


c 


c 


c 


C 


27.9 


(2.43) 


508.6 


(1.58) 


Korea 


31.2 


(9.77) 


12.9 


(1.42) 


1.9 


(1.18) 


64.9 


(2.24) 


a 


a 


a 


a 


30.6 


(3.21) 


537.7 


(2.08) 


Luxembourg 


45.3 


(1.95) 


16.6 


(1.31) 


-2.6 


(1.08) 


62.0 


(2.89) 


- 10.4 


(5.11) 


- 0.2 


(0.10) 


33.0 


(2.22) 


435.7 


(2.40) 


Mexico 


32.6 


(1.59) 


7.5 


(0.92) 


0.8 


(0.34) 


27.8 


(0.80) 


-41.9 


(6.36) 


- 1.8 


(0.15) 


17.9 


(1.03) 


473.7 


(1.02) 


Netherlands 


26.6 


(2.04) 


6.0 


(1.52) 


-1.2 


(1.02) 


106.7 


(2.32) 


-11.6 


(5.72) 


1.7 


(0.14) 


15.3 


(1.85) 


484.5 


(2.33) 


New Zealand 


44.2 


(4.15) 


38.9 


(1.82) 


-1.7 


(1.44) 


56.3 


(3.35) 


-12.2 


(3.84) 


0.0 


(0.10) 


44.8 


(2.62) 


496.5 


(2.44) 


Norway 


37.6 


(18.19) 


34.2 


(2.00) 


-3.4 


(1.62) 


31.1 


(4.32) 


-33.4 


(7.52) 


0.4 


(0.25) 


48.3 


(2.56) 


453.2 


(2.87) 


Poland 


73.8 


(4.44) 


29.4 


(1.59) 


-1.8 


(1.21) 


19.4 


(2.99) 


c 


c 


c 


C 


44.2 


(2.41) 


498.9 


(1.89) 


Portugal 


48.9 


(1.71) 


12.0 


(0.94) 


1.0 


(0.64) 


21.3 


(1.33) 


-5.3 


(5.75) 


0.0 


(0.23) 


22.9 


(1.84) 


518.6 


(1.92) 


Slovak Republic 


34.2 


(3.85) 


14.7 


(1.44) 


-3.2 


(0.98) 


64.3 


(6.30) 


c 


c 


c 


c 


39.1 


(2.58) 


483.2 


(2.33) 


Slovenia 


22.8 


(3.41) 


4.8 


(1.28) 


0.0 


(1.25) 


100.2 


(2.74) 


-23.4 


(7.48) 


-0.2 


(0.24) 


27.7 


(2.16) 


452.4 


(1.63) 


Spain 


61.7 


(1.22) 


9.8 


(0.83) 


0.4 


(0.64) 


22.7 


(1.25) 


-29.7 


(2.86) 


0.4 


(0.04) 


18.0 


(1.42) 


511.3 


(1.07) 


Sweden 


63.8 


(6.69) 


31.4 


(1.82) 


-1.3 


(1.04) 


49.0 


(6.55) 


-38.8 


(8.53) 


0.3 


(0.34) 


43.2 


(2.41) 


454.4 


(3.62) 


Switzerland 


45.5 


(2.75) 


18.2 


(1.27) 


-1.0 


(1.23) 


59.5 


(2.95) 


-25.1 


(3.99) 


-0.7 


(0.11) 


27.0 


(2.00) 


488.8 


(1.50) 


Turkey 


33.7 


(1.96) 


7.7 


(1.50) 


0.3 


(0.61) 


46.3 


(1.70) 


c 


c 


c 


c 


27.9 


(1.74) 


524.0 


(1.59) 


United Kingdom 


35.9 


(6.21) 


27.7 


(2.01) 


-0.3 


(1.51) 


65.7 


(2.49) 


-13.6 


(8.49) 


-0.3 


(0.13) 


23.1 


(2.48) 


468.7 


(1.73) 


United States 


36.3 


(2.17) 


23.5 


(1.70) 


4.4 


(1.15) 


50.4 


(2.56) 


-5.6 


(5.57) 


0.8 


(0.14) 


25.4 


(2.36) 


463.5 


(2.01) 


„ Albania 


11.9 


(5.07) 


20.8 


(3.04) 


3.2 


(1.35) 


43.0 


(2.47) 


c 


c 


c 


c 


56.5 


(3.40) 


421.5 


(3.44) 


| Argentina 


33.6 


(2.50) 


11.2 


(1.96) 


0.9 


(0.87) 


52.6 


(2.03) 


-27.0 


(10.55) 


0.5 


(0.20) 


24.0 


(2.38) 


439.7 


(2.32) 


J Azerbaijan 


13.2 


(1.78) 


10.5 


(1.67) 


1.3 


(0.90) 


36.4 


(2.00) 


-9.8 


(12.34) 


-0.3 


(0.49) 


22.6 


(2.16) 


390.9 


(2.12) 


“■ Brazil 


36.1 


(1.23) 


7.7 


(1.54) 


1.3 


(0.57) 


38.3 


(1.25) 


-71.7 


(17.16) 


-0.9 


(0.47) 


20.2 


(1.63) 


445.5 


(1.33) 


Bulgaria 


27.8 


(5.08) 


15.7 


(1.93) 


0.2 


(1.29) 


75.7 


(3.99) 


c 


c 


c 


c 


42.1 


(3.51) 


423.7 


(2.61) 


Colombia 


33.2 


(1.12) 


6.9 


(2.01) 


0.9 


(0.72) 


39.4 


(1.53) 


c 


c 


c 


c 


3.2 


(2.17) 


477.7 


(1.83) 


Croatia 


31.8 


(2.33) 


10.3 


(1.36) 


-4.0 


(0.99) 


75.3 


(2.01) 


-13.0 


(5.71) 


- 0.1 


(0.22) 


31.4 


(2.56) 


472.8 


(1.69) 


Dubai (UAE) 


34.6 


(1.56) 


15.2 


(1.52) 


3.2 


(1.03) 


25.9 


(3.13) 


21.5 


(3.25) 


1.1 


(0.05) 


28.2 


(3.94) 


362.4 


(2.92) 


Hong Kong-China 


33.6 


(2.03) 


-0.9 


(1.70) 


-1.0 


(0.76) 


41.9 


(1.64) 


23.4 


(3.70) 


- 0.4 


(0.06) 


21.9 


(2.42) 


575.8 


(1.83) 


Indonesia 


14.4 


(2.00) 


4.7 


(2.44) 


0.9 


(0.62) 


29.1 


(1.83) 


c 


c 


c 


c 


28.0 


(1.48) 


430.8 


(2.46) 


J ordan 


47.6 


(6.38) 


17.7 


(1.52) 


0.7 


(0.81) 


26.9 


(1.55) 


-11.5 


(7.50) 


-0.2 


(0.20) 


48.1 


(2.73) 


415.5 


(2.04) 


Kazakhstan 


22.2 


(2.42) 


16.2 


(2.12) 


-1.7 


(1.31) 


55.7 


(2.70) 


-12.2 


(6.78) 


0.0 


(0.10) 


38.1 


(2.23) 


411.1 


(1.57) 


Kyrgyzstan 


20.8 


(2.92) 


18.3 


(2.23) 


1.7 


(1.10) 


75.2 


(2.03) 


-23.4 


(21.78) 


3.3 


(0.50) 


46.0 


(2.45) 


345.7 


(1.83) 


Latvia 


43.8 


(3.07) 


16.2 


(1.89) 


-0.8 


(1.35) 


37.0 


(2.77) 


c 


c 


c 


C 


38.9 


(2.36) 


479.6 


(1.77) 


Liechtenstein 


23.8 


(7.40) 


2.1 


(4.18) 


-5.3 


(3.07) 


112.5 


(12.17) 


-12.6 


(10.22) 


-0.7 


(0.44) 


20.3 


(6.86) 


499.8 


(8.42) 


Lithuania 


27.4 


(2.87) 


18.1 


(1.56) 


0.2 


(1.04) 


44.0 


(2.45) 


c 


c 


c 


c 


51.1 


(2.34) 


447.6 


(1.87) 


Macao-China 


36.7 


(1.01) 


1.8 


(1.61) 


-1.1 


(0.78) 


1.0 


(4.75) 


16.7 


(2.17) 


- 0.1 


(0.23) 


14.1 


(1.51) 


511.0 


(3.47) 


Montenegro 


22.9 


(3.44) 


12.1 


(1.38) 


-0.3 


(1.05) 


64.2 


(6.54) 


-1.8 


(6.69) 


- 1.2 


(0.32) 


39.3 


(2.63) 


409.5 


(2.58) 


Panama 


32.6 


(3.41) 


7.9 


(2.42) 


1.2 


(0.79) 


45.8 


(2.60) 


-3.4 


(10.77) 


- 1.4 


(0.16) 


15.8 


(4.48) 


431.3 


(3.22) 


Peru 


27.5 


(1.23) 


10.5 


(2.05) 


0.9 


(0.64) 


47.2 


(1.46) 


c 


c 


c 


c 


8.3 


(2.17) 


445.6 


(1.59) 


Qatar 


30.7 


(1.70) 


5.3 


(0.98) 


0.4 


(0.85) 


12.7 


(2.91) 


31.5 


(2.98) 


1.7 


(0.07) 


31.4 


(3.71) 


302.5 


(2.94) 


Romania 


19.6 


(4.19) 


10.7 


(1.63) 


-0.3 


(0.79) 


63.9 


(2.34) 


c 


c 


c 


C 


13.7 


(2.56) 


446.4 


(1.70) 


Russian Federation 


31.0 


(2.01) 


18.2 


(1.93) 


-1.6 


(1.40) 


38.8 


(3.32) 


-9.1 


(5.88) 


-0.4 


(0.22) 


38.7 


(2.28) 


452.9 


(1.89) 


Serbia 


21.3 


(4.48) 


9.2 


(1.25) 


-0.8 


(0.74) 


55.1 


(3.42) 


1.2 


(5.65) 


0.3 


(0.13) 


27.1 


(2.22) 


425.1 


(1.60) 


Shanghai-China 


21.8 


(3.34) 


4.6 


(1.41) 


0.1 


(0.85) 


57.3 


(1.48) 


c 


c 


c 


c 


29.3 


(1.98) 


583.5 


(2.04) 


Singapore 


28.9 


(2.09) 


22.2 


(2.19) 


-2.8 


(1.14) 


104.7 


(2.86) 


0.4 


(4.21) 


- 1.0 


(0.13) 


24.6 


(2.57) 


590.2 


(2.76) 


Chinese Taipei 


15.4 


(4.12) 


15.5 


(1.50) 


-1.2 


(1.05) 


82.8 


(3.06) 


c 


c 


C 


c 


36.8 


(2.25) 


515.6 


(2.03) 


Thailand 


22.1 


(2.05) 


10.4 


(1.54) 


2.4 


(0.66) 


28.8 


(1.31) 


a 


a 


a 


a 


31.3 


(1.78) 


454.6 


(1.67) 


Trinidad and Tobago 


35.3 


(1.60) 


-0.6 


(2.00) 


-0.2 


(0.91) 


123.2 


(3.42) 


- 9.2 


(13.59) 


- 0.7 


(10.28) 


40.4 


(2.90) 


484.9 


(2.77) 


Tunisia 


49.7 


(1.57) 


3.7 


(1.76) 


0.7 


(0.56) 


17.8 


(1.25) 


c 


c 


c 


c 


14.4 


(1.84) 


449.6 


(1.63) 


Uruguay 


41.4 


(1.49) 


12.4 


(1.58) 


0.5 


(0.75) 


29.7 


(1.58) 


c 


c 


c 


c 


30.1 


(2.48) 


464.2 


(2.29) 



StatLink http://dx.doi.org/10.n87/88s932343m 
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Appendix G. Explanation of Indices (abstracted from OECD 2010c and OECD 2010cl) 



Economic, social and cultural status 

The PISA index of economic, social and cultural status (ESCS) was derived from the following three indices: highest occupational status of parents (HIS El), 
highest educational level of parents in years of education according to ISCED (PARED), and home possessions (HOM EPOS). The index of home possessions 
(HOMEPOS) comprises all items on the indices of WEALTH , CULT POSS and HEDRES, as well as books in the home recoded into a four-level categorical 
variable (0-10 books, 11-25 or 26-100 books, 101-200 or 201-500 books, more than 500 books). 

The PISA index of economic, social and cultural status (ESCS) was derived from a principal componentanalysis of standardized variables (each variable has an 
OECD mean of zero and a standard deviation of one), taking the factor scores for the first principal component as measures of the index of economic, social and 
cultural status. 

Principal componentanalysis was also performed foreach participating country to determine to whatextentthe components of the index operate in similarways 
across countries. The analysis revealed that patterns of factor loading were very similar across countries, with all three components contributing to a similar 
extentto the index. Forthe occupational component the average factor loading was 0.80, ranging from 0.66 to 0.87 across countries. Forthe educational 
component the average factor loading was 0.79, ranging from 0.69 to 0,87 across countries. Forthe home possession component the average factor loading 
was 0.73, ranging from 0.60 to 0.84 across countries. The reliability of the index ranged from 0.41 to 0.81. These results support the crossnational validity of the 
PISA index of economic, social and cultural status. 

The imputation of components for students missing data on one componentwas done on the basis of a regression on the other two variables, with an additional 
random error component The final values on the PISA index of economic, social and cultural status (ESCS) have an OECD mean of 0 and a standard deviation 
of 1. 

Enjoyment of reading activities 

The index of enjoyment of reading (J OYREAD) activities was derived from students' level of agreement with the following statements (ST24): i) I read only if I 
have to; ii) reading is one of my favorite hobbies; iii) I like talking about books with other people; iv) I find it hard to finish books; v) I feel happy if I receive a book 
as a present; vi) forme, reading is a waste of time; vli) I enjoy going to a bookstore ora library; viii) I read only to get information that I need; ix) I cannot sit still 
and read for more than a few minutes; x) I like to express my opinions about books I have read; and xi) I like to exchange books with my friends. 

As all items thatare negatively phrased (items i, iv, vi, viii and ix) are inverted for scaling, the highervalues on this index indicate higher levels ofenjoymentof 
reading. 

Diversity of reading materials 

The index of diversity of reading materials (DIVREAD) was derived from the frequency with which students read the following materials because they want to 
(ST25): magazines, comic books, fiction, non-fiction books and newspapers. The highervalues on this index indicate higher diversity in reading. 

Online reading activities 

The index of online reading activities (ONLNREAD) was derived from the frequency with which students involved in the following reading activities (ST26): 
reading emails, <chat on line>, reading online news, using an online dictionary orencyclopedia, searching online information to learn abouta particular topic, 
taking part in online group discussions or forums and searching for practical information online. The highervalues on this index indicate more frequent online 
reading activities. 

Approaches to learning 

How students approach learning is based on student responses in ST27 and measured through the following three indices: memorization (MEMOR), elaboration 
(E LAB) and control strategies (CSTRAT). 

The index of memorization (MEMOR) was derived from the frequency with which students did the following when they were studying: i) try to memorize 
everything that is covered in the text; ii) try to memorize as many details as possible; iii) read the text so many times that they can recite it; and iv) read the text 
over and over again. 

The index of elaboration (ELAB) was derived from the frequency with which students did the following when they were studying: i) try to relate new information to 
prior knowledge acquired in other subjects; ii) figure out how the information might be useful outside school; iii) try to understand the material better by relating it 
to my own experiences; and iv) figure out how the text information fits in with what happens in real life. 

The index of control strategies (CSTRAT) was derived from students' reports on how often they did the following statements: i) when I study, I start by figuring out 
what exactly I need to learn; ii) when I study, I check if I understand what I have read; iii) when I study, I try to figure out which concepts I still haven't really 
understood; iv) when I study, I make sure that I remember the most important points in the text; and v) when I study and I don't understand something, I look for 
additional information to clarify this. Highervalues on the index indicate higher importance attached to the given strategy. 

Attitudes towards school 

The index of attitude towards school (ATSCHL ) was derived from students' level ofagreementwith the following statements in ST33: i) school has done little to 
prepare me for adult life when I leave school; ii) school has been a waste of time; iii) school has helped give me confidence to make decisions; iv) school has 
taughtme things which could be useful in a job. As all items thatare negatively phrased i) and ii) are inverted forscaling, highervalues on this index indicate 
perception of a more positive school climate. 
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Appendix G. Explanation of Indices (abstracted from OECD 2010c and OECD 2010cl) 



Teacher-student relations 

The index of teacher-student relations (STUDREL) was derived from students' level of agreement with the following statements in ST34: i) I get along well with 
most of my teachers; ii) most of my teachers are interested in my well-being; Hi) most of my teachers really listen to what I have to say; iv) if I need extra help, I 
will receive it from my teachers; and v) most of my teachers treat me fairly. Higher values on this index indicate positive teacher-student relations. 

Disciplinary climate 

The index of disciplinary climate (DISCLIMA) was derived from students' reports on how often the followings happened in their lessons of the language of 
instruction (ST36): i) students don't listen to what the teacher says; ii) there is noise and disorder; iii) the teacher has to wait a long time for the students to <quiet 
down>; iv) students cannot work well; and v) students don't start working fora long time after the lesson begins. As all items are inverted for scaling, higher 
values on this index indicate a better disciplinary climate. 

Teachers' stimulation of students' reading engagement 

The index of teachers' stimulation of students' reading engagement (STIM READ) was derived from students' reports on how often the following occurred in their 
lessons of the language of instruction (ST37): i) the teacher asks students to explain the meaning of a text; ii) the teacher asks questions that challenge students 
to get a better understanding of a text; iii) the teacher gives students enough time to think about their answers; iv) the teacher recommends a book or author to 
read; v) the teacher encourages students to express their opinion about a text; vi) the teacher helps students relate the stories they read to their lives; and vii) the 
teacher shows students how the information in texts builds on what they already know. Higher values on this index indicate higher teachers' stimulation of 
students' reading engagement. 

Use of structuring and scaffolding strategies 

The index of use of structuring and scaffolding strategies (STRSTRAT) was derived from students reports on how often the following occurred in their lessons of 
the language of instruction (ST38): i) the teacher explains beforehand what is expected of the students; ii) the teacher checks that students are concentrating 
while working on the -creading assignment*; iii) the teacher discusses students' work, after they have finished the <reading assignment*; iv) the teacher tells 
students in advance how their work is going to be judged; v) the teacher asks whether every student has understood how to complete the -creading assignment*; 
vi) the teacher marks students' work; vii) the teacher gives students the chance to ask questions about the -creading assignment^ viii) the teacher poses 
questions that motivate students to participate actively; and ix) the teacher tells students how well they did on the -creading assignment immediately after. 
Highervalues on this index indicate a greater use of structured teaching. 

Use of libraries 

The index of use of libraries (LIB USE) was derived from students' reports on the frequency for visiting a library for the following activities (ST39): i) borrow books 
to read for pleasure; ii) borrow books for school work; iii) work on homework, course assignments or research papers; iv) read magazines or newspapers; v) read 
books forfun; vi) learn about things that are not course-related, such as sports, hobbies, people ormusic; and vii) use the Internet. Highervalues on this index 
indicate a greatuse of libraries. 

Metacognition strategies: understanding and remembering 

The index of understanding and remembering (UNDREM) was derived from students' reports on the usefulness of the following strategies for understanding and 
memorizing the text (ST41): A) I concentrate on the parts of the text that are easy to understand; B) I quickly read through the text twice; C) After reading the text, 
I discuss its content with other people; D) I underline important parts of the text; E) I summarize the text in my own words; and F) I read the text aloud to another 
person. 

This index was scored using a rater-scoring system. Through a variety of trial activities, both with reading experts and national centers, a preferred ordering of 
the strategies according to their effectiveness to achieve the intended goal was agreed. The experts' agreed orderof the six items consisting this index is CDE > 
ABF. Scaling was conducted with two steps. First, a score was assigned to each student, which is a number that ranged from 0 to 1 and can be interpreted as 
the proportion of the total number of expert pair-wise relations that are consistent with the student ordering. For example, if the expert rule is (ABFD>CEG, 

4 '3=12 pairwise rules are created (i.e. A>C, A>E, A>G, B>C, B>E, B>G, F >C , F >E , F >G , D>C, D>E, D>G). If the responses of a studenton this task follow 8 of 
the 12 rules, the student gets a score of 8/12 =0.67. Second, these scores were standardized for the index to have a mean of 0 and a standard deviation of 1 
across OECD countries. Highervalues on this index indicate greater students' perception of usefulness of this strategy. 

Metacognition strategies: summarizing 

The index of summarizing (METASUM) was derived from students' reports on the usefulness of the following strategies forwriting a summary of a long and 
rather difficult two-page text about fluctuations in the water levels of a lake in Africa (ST42): A) I write a summary. Then I check that each paragraph is covered in 
the summary, because the contentofeach paragraph should be included; B) I try to copy out accurately as many sentences as possible; C) before writing the 
summary, I read the textas many times as possible; D) I carefully check whether the most importantfacts in the textare represented in the summary; and E) I 
read through the text, underlining the most important sentences, then I write them in my own words as a summary. This index was scored using a rater-scoring 
system. The experts' agreed orderof the five items consisting this index is DE>AC>B. Highervalues on this index indicate greater students' perception of 
usefulness of this strategy. 

Reading for school 

Students' engagement in reading for school is based on student responses to 17 items included in the last page of the test booklets and measured through the 
following four indices: index of interpretation of literary texts (RFSINTRP), index of use of texts containing non-continuous materials (RFSNCONT), index of 
reading activities for traditional literature courses (RFSTRLIT), index of use of functional texts (RFSFUMAT). 

For each item students were asked to report whether they read different texts forschool (either in the classroom or as homework) "many times", 'two orthree 
times", "once", or "not at all". All items are inverted forscaling, so that higher values on this index indicate higher levels of enjoyment of reading. 

The index of interpretation of literary texts (RFSINTRP) was derived from the frequency with which students reported that in the past month they did the following: 
i) read fiction; ii) explain the cause of events in a text; iii) explain the way characters behave in a text; iv) explain the purpose of a text. 
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Appendix G. Explanation of Indices (abstracted from OECD 2010c and OECD 2010d) 



The index of use of texts containing non-cont'nuous materials (RFSNCONT) was derived from the frequency with which students reported that in the past month 
they did the following: i) use texts that include diagrams or maps; ii) use texts that include tables or graphs: iii) find information from a graph, diagram or table; 
and iv) describe the way the information in a table orgraph is organized. 

The index of reading activities for traditional literature courses (R F STRUT) was derived from the frequency with which students reported that in the past month 
they did the following: i) read information texts about writers or books; ii) read poetry; iii) memorize a text by heart; iv) learn about the place of a text in the history 
of literature; v) learn about the life of the writer. 

The index of use of functional texts (RFSFUMAT) was derived from the frequency with which students reported that in the past month they did the following: i) 
read newspaper reports and magazine articles; ii) read instructions or manuals telling how to make or do something (e.g. how a machine works); and iii) read 
advertising material (e.g. advertisements in magazines, posters). 

School-level simple indices 

School and class size 

The index of school size (SC FI SIZE) was derived by summing up the number of girls and boys at a school (SC 06). 

Student-teacher ratio 

Student-teacher ratio (STRATIO) was obtained by dividing the school size by the total number of teachers. The number of part-time teachers (SC09Q12) was 
weighted by 0.5 and the numberof full-time teachers (SC09Q11) was weighted by 1.0 in the computation of this index. 

Proportion of girls enrolled at school 

The index of the proportion of girls in the school (PCGIRLS) was derived from the enrolment data (SC06). 

School type 

Schools are classified into as either public or private, according to whether a private entity ora public agency has the ultimate power to make decisions 
concerning its affairs (SC02). This information is combined with SC03 which provides information on the percentage of total funding which comes from 
government sources to create the index of school type (SCHTYPE). This index has three categories: (1) public schools controlled and managed by a public 
education authority or agency, (2) government-dependent private schools controlled by a non-government organization or with a governing board not selected by 
a government agency that receive more than 50% of their core funding from government agencies, (3) government-independent private schools controlled by a 
non-government organization or with a governing board not selected by a government agency that receive less than 50% of their core funding from government 
agencies. 

Availability of computers 

The index of computer availability (IRATCOMP) was derived from dividing the numberof computers available for educational purposes available to students in 
the modal grade for 15-year-olds (SC10Q02) by the numberof students in the modal grade for 15-year-olds (SC10Q01). 

The index of computers connected to the Internet (COMP WEB) was derived from dividing the numberof computers for educational purposes available to 
students in the modal grade for 15-year-olds that are connected to the web (SC10Q03) by the numberof computers for educational purposes available to 
students in the modal grade for 15-year-olds (SC10Q02). 

Quantity of teaching staff at school 

The proportion of fully certified teachers (PROPCERT) was computed by dividing the numberof fully certified teachers (SC09Q21 plus 0.5*SC09Q22) by the total 
numberof teachers (SC09Q11 plus 0.5*SC 09Q 12). The proportion of teachers who have an ISCED 5A qualification (PROPQUAL) was calculated by dividing the 
numberof these kind of teachers (SC09Q31 plus 0.5*SC09Q32) by the total numberof teachers (SC09Q11 plus 0.5*SC09Q12). 

Academic selectivity 

The index of academic selectivity (SELSCH) was derived from school principals' responses on how frequently consideration was given to the following factors 
when students were admitted to the school, based on a scale from the response categories "never", "sometimes" and "always" (SC19Q02 and SC19Q03): 
student's record of academic performance (including placement tests); and recommendation of feeder schools. This index has the following three categories: (1) 
schools where these two factors are "never" considered for student admittance, (2) schools considering at least one of these two factors "sometimes" but neither 
factor "always", and (3) schools where at least one of these two factors is "always" considered for student admittance. 

Ability grouping 

The index of ability grouping between classes (ABGROUP) was derived from the two items of school principals' reports on whether school organises instruction 
differently for student with different abilities 'for all subjects", 'for some subjects", or "not for any subject" (SC 12Q01 for grouping into different classes and 
SC12Q02 for grouping within classes). This index has the following three categories: (1) schools that do notgroup students by ability in any subjects, either 
between or within classes; (2) schools that group students by ability for some, but not all, subjects, and that do so either between or within classes; and (3) 
schools thatgroup students by ability in all subjects either between orwithin classes. 

School-level scale indices 

School responsibility for resource allocation 

School principals were asked to report whether "principals", "teachers", "school governing board", "regional or local education authority" or "national education 
authority” has a considerable responsibility for the following tasks (SC 24): i) selecting teachers for hire; ii) dismissing teachers; iii) establishing teachers' starting 
salaries; iv) determining teachers' salaries increases; v) formulating the school budget; and vi) deciding on budget allocations within the school. The index of 



2009 Data Files and Database with U.S. Specific Variables 



G-9 




Appendix G. Explanation of Indices (abstracted from OECD 2010c and OECD 2010d) 



school responsibility for resource allocation (RESPRES) was derived from these six items. The ratio of the number of responsibility that "principals" and/or 
"teachers" have for these six items to the number of responsibility that "regional or local education authority" and/or "national education authority” have for these 
six items was computed. Positive values on this index indicate relatively more responsibility for schools than local, regional or national education authority. This 
index has an OECD mean of 0 and a standard deviation of 1. 

School responsibility for curriculum and assessment 

School principals were asked to report whether "principals”, "teachers", "school governing board", "regional or local education authority", or "national education 
authority” has a considerable responsibility for the following tasks (SC24): i) establishing student assessment policies; ii) choosing which textbooks are used; iii) 
determining course content; and iv) deciding which courses are offered. The index of the school responsibility for curriculum and assessment (RE SPC UR R) was 
derived from these four items. The ratio of the number of responsibility that "principals” and/or "teachers" have for these four items to the number of responsibility 
that "regional or local education authority” and/or "national education authority” have for these four items was computed. Positive values on this index indicate 
relatively more responsibility for schools than local, regional or national education authority. This index has an OECD mean of 0 and a standard deviation of 1. 

Teacher participation 

The index of teacher participation (TCH PARTI) was scaled based on all 12 items in SC 24 using school principals' responses that "teachers" have considerable 
responsibility. Higher values on this index indicate greater teachers' participation. 

School principal's leadership 

The index of school principal's leadership (LDRSHP) was derived from school principals' responses about the frequency with which they were involved in the 
following school affairs in the previous school year (SC26): i) make sure thatthe professional development activities of teachers are in accordance with the 
teaching goals of the school; ii) ensure thatteachers work according to the school's educational goals; iii) observe instruction in classrooms; iv) give teachers 
suggestions as to how they can improve their teaching; v) use student performance results to develop the school's educational goals; vi) monitor students' work; 
vii) take the initiative to discuss matters, when a teacher has problems in his/her classroom; viii) inform teachers about possibilities for updating their knowledge 
and skills; ix) check to see whether classroom activities are in keeping with our educational goals; x) take exam results into account in decisions regarding 
curriculum development; xi) ensure that there is clarity concerning the responsibility for coordinating the curriculum; xii) solve the problem together, when a 
teacher brings up a classroom problem; xiii) pay attention to disruptive behavior in classrooms; and xiv) take over lessons from teachers who are unexpectedly 
absent Highervalues on this index indicate greater involvement of school principals in school affairs. 

Teacher shortage 

The index of teacher shortage (TCSHORT) was derived from four items measuring school principals' perceptions of potential factors hindering instruction at their 
school (SC 11). These factors are a lack of: i) qualified science teachers; ii) a lack of qualified mathematics teachers; iii) qualified <test languages teachers; and 
iv) qualified teachers of other subjects. Highervalues on this index indicate school principals' reports of higher teacher shortage ata school. 

School's educational resources 

The index on the school's educational resources (SC MATE DU) was derived from seven items measuring school principals' perceptions of potential factors 
hindering instruction at their school (SC 11). These factors are: i) shortage or inadequacy of science laboratory equipment; ii) shortage or inadequacy of 
instructional materials; iii) shortage or inadequacy of computers for instruction; iv) lack or inadequacy of Internet connectivity; v) shortage or inadequacy of 
computer software for instruction; vi) shortage or inadequacy of library materials; and vii) shortage or inadequacy of audio-visual resources. As all items were 
inverted forscaling, highervalues on this index indicate better quality of educational resources. 

Extra-curricular activities offered by school 

The index of extra-curricular activities (EXCURACT) was derived from school principals' reports on whether their schools offered the following activities to 
students in the national modal grade for 15-year-olds in the academic year of the PISA assessment (SC 13): i) band, orchestra or choir; ii) school play or school 
musical; iii) school yearbook, newspaper or magazine; iv) volunteering or service activities; v) bookclub; vi) debating club or debating activities; vii) school club or 
school competition for foreign language mathematics or science; viii) <academic club>; ix) art club or art activities; x) sporting team or sporting activities; xi) 
lectures and/or seminars; xii) collaboration with local libraries; xiii) collaboration with local newspapers; and xiv) ccountry specific item>. Highervalues on the 
index indicate higher levels ofextra-curricularschool activities. 

Teacher behavior 

The index on teacher-related factors affecting school climate (TEACBEHA) was derived from school principals' reports on the extent to which the learning of 
students hindered by the following factors in their schools (SC17): i) teachers' low expectations of students; ii) poor student-teacher relations; iii) teachers not 
meeting individual students' needs; iv) teacher absenteeism; v) staff resisting change; vi) teachers being too strict with students; and vii) students not being 
encouraged to achieve theirfull potential. As all items were inverted forscaling, highervalues on this index indicate a positive teacher behavior. 

Student behavior 

The index of student-related factors affecting school climates (STUDBEHA) was derived from school principals' reports on the extent to which the learning of 
students hindered by the following factors in their schools (SC17): i) student absenteeism; ii) disruption of classes by students; iii) students skipping classes; iv) 
student lacking respect for teachers; v) student use of alcohol or illegal drugs; and vi) students intimidating or bullying other students. As all items were inverted 
forscaling highervalues on this index indicate a positive student behavior. 

Parent questionnaire simple indices 

Educational level of parents 

The educational level of parents is classified using ISCED (OECD, 1999) based on parents' responses (PA09 and PA10). Three indices were constructed: 
educational level for mother (PQMISCED); educational level for father (PQ FISC ED); and the highest educational level of parents (PQH ISCED), which 
corresponds to the higher ISCED level of either pa rent. These indices have the following categories: (0) None, (1) ISCED 3A (upper secondary), (2) ISCED 4 
(non-tertiary post-secondary), (3) ISCED 5B (vocational tertiary), and (4) ISCED 5A, 6 (theoretically oriented tertiary and post-graduate). 
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Parent questionnaire scale indices 

Parents' perception of school quality 

The index of parents' perception of school quality (PQSCHOOL) was derived from parents' level of agreement with the following statements (PAW): i) most of my 
child's school teachers seem competentand dedicated; ii) standards of achievementare high in my child's schools; iii) I am happy with the content taught and 
the instructional methods used in my child's school; iv) I am satisfied with the disciplinary atmosphere in my child's school; v) my child's progress is carefully 
monitored by the school; vi) my child's school provides regular and useful information on my child's progress; and vii) my child's school does a good job in 
educating students. As all items were inverted forscaling, highervalues on this index indicate parents' positive evaluations of the school's quality. 

Parents' involvement in school 

The index of parents' involvement in school (PAR INVOL) was derived from parents' responses to whether they have participated in various school-related 
activities during the previous academic year (PA15). Parents were asked to report "yes" or "no" for the following statements: i) discuss my child's behavior or 
progress with a teacher on my own initiative; ii) discuss my child's behavior or progress on the initiative of one of my child's teachers; iii) volunteer in physical 
activities; iv) volunteer in extracurricular activities; v) volunteer in school library or media centre; vi) assist a teacher in school; vii) appear as a guest speaker; and 
viii) participate in local school. Highervalues on this index indicate greater parents' involvementin school. 

Students reading resources at home 

The index of students' reading resources at home (READRES) was derived from parents' reports on whether the followings are available for their children in their 
home (PA07): i) email; ii) online chat; iii) Internet connection; iv) daily newspaper; v) subscription to journal or magazine; and vi) books of his/her own (not school 
books). Highervalues on this index indicate greater availability of reading resources at home. 

Parents' current support of their child's reading literacy 

The index of parents' current support of their child's reading literary (CURSUPP) was derived from parents' reports on the frequency with which they or someone 
else in their home did the following with their child (PA08): i) discuss political or social issues; ii) discuss books, films or television programs; iii) discuss how well 
the child is doing at school; iv) go to a bookstore or library with the child; v) talk with the child about what he/she is reading; and vi) help the child with his/her 
homework. Highervalues on this index indicate greater parental support of child's reading literacy. 

Parents' support of their child's reading literacy at the beginning of primary school 

The index of parents' support of their child's reading literacy at the beginning of primary school (PRESUPP) was derived from parents' reports on the frequency 
with which they or someone else in their home undertook the following activities with their child when the child attended the first year of primary school (PA03): i) 
read books; ii) tell stories; iii) sing songs; iv) play with alpha bet toys; v) talk about what parent had read; vi) play word games; vii) wrote letters or words; and viii) 
read aloud signs and labels. Highervalues on this index indicate greater levels of parents' support 

Motivational attributes of parents' own reading engagement 

The index of motivational attributes of parents' own reading engagement (MOT READ) was derived from parents' level of agreement with the following 
statements (PA06): i) reading is one of my favorite hobbies; ii) I feel happy if I receive a book as a present; iii) for me reading is a waste of time; and iv) I enjoy 
going to a bookstore or library. As the item iii was inverted forscaling, highervalues on this index indicate greater parents' motivation to engage in reading 
activities. 
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Nonresponse Bias Analysis Report 



H.1 Introduction 

An important component of survey or assessment data quality is the representativeness of the study 
sample. This representativeness of the population is achieved by selecting a sample of respondents who 
are similar to the population in terms of key demographic variables. However, in practice not all sampled 
respondents participate in surveys. If enough respondents fail to participate or if respondents differ in 
their response rates by key demographic characteristics, the final sample may not represent the target 
population. The extent to which the distributions of the sampled respondents differ from the 
corresponding distributions of the population is termed unit nonresponse bias. One way to characterize and 
quantify the presence of unit nonresponse bias is to compare responding sample elements with 
nonresponding sample elements with respect to underlying sociodemographic characteristics for which 
data are available on the frame. In the case of Program for International Student Assessment (PISA) 
2009, unit nonresponse can occur at both the school and student levels. 

Also, not all sample respondents respond to all applicable items in the questionnaire. If enough 
respondents fail to respond to a question or if respondents differ in their item response rates by key 
demographic characteristics, the item respondents may not represent the student respondents. The 
extent to which the distributions of the item respondents differ from the corresponding distributions of 
the sample respondents is termed item nonresponse bias. One way to characterize and quantify the presence 
of item nonresponse bias is to compare responding item cases with nonresponding item cases with 
respect to underlying sociodemographic characteristics for which data are available on the frame or on 
other items on the questionnaire. 

The National Center for Education Statistics (NCES) standards for assessment surveys stipulate that a 
nonresponse bias analysis is required at any stage of data collection with a weighted unit response rate 
less than 85 percent. The international PISA 2009 standards also require that nonresponse bias analyses 
need to be conducted if school response rates are less than 85 percent. For PISA 2009, the weighted 
school response rate for originally sampled schools was 68 percent 11 and the weighted student response 
rate was 87 percent. Thus, unit nonresponse bias analysis was required at the school level, but not at the 
student level. Some questionnaire items also had response rates below 85 percent, and thus this report 
includes analyses of those items. 

The objective of this nonresponse bias analysis is to shed light on any biases that might be present in the 
data because of nonresponse. That is, responding and nonresponding schools are compared using 
information from the sampling frame to determine whether responding schools are representative of the 
original sample or whether there are significant differences between the responding and nonresponding 
schools. Also, respondents and nonrespondents to questionnaire items are compared using information 
from the sampling frame and other questionnaire items to determine whether students responding to 
items actually represent a virtually random subsample of the sample respondents or whether there are 
significant differences between the students responding and not responding to items. 



11 Response rates reported here are based on the formula used in the international report and are not consistent with 
NCES standards. A more conservative way to calculate the response rate would be to include replacement schools that 
participated in the denominator as well as the numerator, and to add replacement schools that were hard refusals to the 
denominator. This results in a response rate of 64 percent. 
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There are two major categories of analyses: (1) analyses of school nonresponse bias; and (2) analyses of 
nonresponse bias for individual items on the School Questionnaire and Student Questionnaire. 

H.2 Summary of PISA 2009 Sampling in the United States 

The PISA 2009 U.S. sample for the main study was selected using a two-stage design — a sample of 
schools and a sample of students within sampled schools. The two-stage sample design was 
implemented to attain an approximately self-weighting sample of students where each 15-year-old 
student in the United States had an equal probability of being selected for the study. 

Eligible schools in the PISA 2009 school frame included 67,309 schools. These included schools with 
grade 7 or higher operating in the 50 states and the District of Columbia, ungraded schools, Department 
of Defense (DoD) domestic schools, Bureau of Indian Education (BIE) schools, special education 
schools, and vocational education schools. Schools were stratified by Census region (Northeast, 
Midwest, South, and West) and school type (public and private). Within each stratum, schools were 
sorted by grade range, locality, first three digits of the zip code, high/low minority percentage, and 
student enrollment. A systematic sample was selected independently in each stratum. The selection 
probability for each school was proportional to a measure of size. The student population for the 2009 
PISA was defined by age to include all students between 1 5 years and 3 (completed) months and 1 6 years 
and 2 (completed) months at the start of the testing period. The enrollment of PISA-eligible students 
(ENR) in each school was estimated using school enrollment and the percentage of age eligible students 
per grade. 

The U.S. school sample included 236 schools — 201 large schools with at least 42 estimated eligible 
students, 10 moderately small schools with between 21 and 45 estimated eligible students, and 25 very 
small schools with less than 21 estimated eligible students. In addition, for each school selected in the 
sample, two neighboring schools in the sampling frame were designated as substitute schools. 

In order to achieve the required student yield of 35 assessed students per school (taking into account 
student exclusions and absences), the United States set a target cluster size (TCS) of 42 students per 
school. School coordinators were asked to generate lists of 1 5-year-old students (defined as students 
with birthdates between July 1, 1993 and June 30, 1994). A total of 56,221 students were listed from the 
165 participating schools. The average list size was 340 students. A total of 6,677 students (an average 
of 40.5 per school) were randomly sampled. 

Additional details about PISA sampling are presented in Chapter 3 of this User’s Guide. 

H.3 Methodology for Analysis of School Nonresponse Bias 

Three types of comparisons were performed to examine the effects of school nonresponse: 

1 . A comparison of participating originally sampled schools (n=145) with all eligible originally sampled schools 
(n=208) using school-level weights after trimming but before nonresponse adjustments; 

2. A comparison of participating schools (145 original and 20 replacement schools) (n=165) with all 
sampled eligible schools (n=208) using school-level weights after trimming but before nonresponse 
adjustments; and 

3. A comparison of participating schools (145 original and 20 replacement schools) (n=165) with all 
sampled eligible schools (n=208) using school-level weights after nonresponse adjustments. 



2009 Data Files and Database with U.S. Specific Variables 



H-3 



Appendix H. Nonresponse Bias Analysis Report 



Eligible originally sampled schools (n=208) for the first set of analyses were those that: (1) were eligible 
for PISA 2009 data collection (28 originally sampled schools had either closed or had no PISA-eligible 
students); (2) were selected in the original sample frame; and (3) either participated or refused to 
participate. Sampled eligible schools (n=208) for the second and third sets of analyses consisted of the 
20 participating replacement schools and 188 originally sampled schools that: (1) were eligible for PISA 
2009 data collection; (2) were replacement schools or originally sampled schools that were not replaced; 
and (3) either participated or refused to participate. 

All of the analyses were done using weights provided by Westat, an ACER subcontractor (see Section 
8.3 of this report for a further description of weighting). The first and second sets of analyses used 
school-level weights after trimming but before nonresponse adjustments. The third set of analyses used 
weights that had been adjusted for nonresponse. PISA’s approach to nonresponse weighting involves 
weight adjustments within sampling strata based on response rates within the strata. Thus, PISA does 
not use nonresponse bias analyses or poststratification to create nonresponse adjusted weights. Both 
weights provided by Westat were multiplied by the estimated number of PISA-eligible students in the 
school to decrease the impact of very small schools. Because of the stratified sampling design used for 
PISA, all analyses were done using Wesvar (Westat 2007) in order to generate appropriate error terms. 

The specific variables on which groups were compared came from the sampling frame. The data for 
public schools were from the 2005—06 Common Core of Data (CCD), and the data for private schools 
were from the 2005—06 Private School Universe Survey (PSS). The variables used to compare groups 
included both categorical and continuous variables. The categorical variables were: 

• School control (public/private); 

• Community type (central city, urban area, town, rural area); 12 and 

• Census region (Northeast, Midwest, South, West). 13 

The continuous variables were: 

• Total number of students in the school; 

• Estimated number of age-eligible students in the school; 

• Percentage of White, non- Hispanic students; 

• Percentage of Black, non-Hispanic students; 

• Percentage of Hispanic students; 

• Percentage of Asian/Pacific Islander, non-Hispanic students; 

• Percentage of American Indian/Alaska Native, non-Hispanic students; 

• Percentage of students eligible for free or reduced-price student lunch; 14 and 



12 Community types are defined as follows: (1) a “central city” is the principal city in a Metropolitan Core Based 
Statistical Area (MCBSA) as defined by the U.S. Census Bureau; (2) an “urban area” is any other area within a MCBSA; 
(3) a “town” is an incorporated place with a population of at least 2,500 and not within a MCBSA; and (4) a “rural area” 
is an area not in a MCBSA and not in a town. 

13 The Northeast region consists of Connecticut, Maine, Massachusetts, New Hampshire, New Jersey, New York, 
Pennsylvania, Rhode Island, and Vermont. The Midwest region consists of Illinois, Indiana, Iowa, Kansas, Michigan, 
Minnesota, Missouri, Nebraska, North Dakota, Ohio, Wisconsin, and South Dakota. The South region consists of 
Alabama, Arkansas, Delaware, the District of Columbia, Florida, Georgia, Kentucky, Louisiana, Maryland, Mississippi, 
North Carolina, Oklahoma, South Carolina, Tennessee, Texas, Virginia, and West Virginia. The West region consists of 
Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oregon, Utah, Washington, and 
Wyoming. 
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• Percentage of male students. 

For categorical variables, a Rao-Scott (RS2) modified chi-square statistic was used to test differences 
between participating and eligible sampled schools. For details of how this statistic is calculated in 
Wesvar, see Westat (2007). 

For continuous variables, summary means for the two groups were calculated and the difference in 
means tested using / tests: 



K-V.4 - Vfi)l 

SCK-Y.4 -X B ) 



where Xj and X B are the means for the two groups and - X B ) is the standard error of the 

difference between means taking into account the complex survey design. 

In addition, in order to examine the combined effect of these variables on school participation, logistic 
regressions were performed using the variables described above: 



log 



i-P, 



=fi 0 + ^U + ^ 2t + ... +fiX pl 



where X^, X>,-, . . . , X pi are p variables associated with a given case, and /q, ••■■>f p are coefficients 
to be estimated. 



H.4 School Nonresponse Bias Among Originally Sampled Schools — 
Nonadjusted Weights 



As described in Section H3, the first set of analyses compared participating originally sampled schools 
(n=145) with all eligible originally sampled schools (n=208). The first group of analyses of this type 
consisted of chi-square tests of the differences between participating originally sampled schools and 
eligible originally sampled schools on categorical variables. Tables Id-1 through Table H-3 show the 
results for the variables school control (public/private), community type (central city, urban area, town, 
rural area), and Census region (Northeast, Midwest, South, West). The tables show: (1) the percentage 
estimates and standard errors for the two groups; (2) the difference in the estimates (bias); (3) the relative 
bias (the bias divided by the estimate for eligible sampled schools); and (4) the probability value for the 
Rao-Scott modified chi-square statistic. 



14 The National School Lunch Program is a federally assisted meal program for school children. If a household’s total 
income is below a certain amount, the children in that household can eat free or at a very-reduced-price. Thus, the 
percentage of students receiving a lunch is an indicator of the socioeconomic level of families served by the school. This 
information was available only for public schools. 
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Table H.l. School control for participating originally sampled schools and eligible originally sampled 
schools 



School characteristic 


Participating originally 
sampled schools 
(n=145) 

Standard 
Percent error 


Eligible originally 
sampled schools 
(n=208) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square test 
p value 


School control 














.773 


Public 


92.9 


1.45 


92.5 


0.41 


0.4 


0.4 




Private 


7.1 


1.45 


7.5 


0.41 


-0.4 


-5.3 




NOTE: Detail may notsum to totals due to rounding. 












SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 


Table H.2. Community type for participating originally sampled schools and eligible originally sampled 


schools 


















Participating originally 


Eligible originally 










sampled schools 


sampled schools 










(n=145) 


(n=208) 












Standard 




Standard 




Relative 


Chi-square test 


School characteristic 


Percent 


error 


Percent 


error 


Bias 


bias 


p value 


Community type 














.899 


Central city 


30.5 


3.84 


30.3 


3.23 


.0.2 


0.6 




Urban area 


33.7 


4.00 


35.0 


3.38 


-1.3 


-3.7 




Town 


13.3 


2.85 


12.4 


2.31 


0.9 


7.3 




Rural area 


22.6 


3.48 


22.2 


2.90 


0.4 


1.8 





NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 



Table H.3. Census region for participating originally sampled schools and eligible originally sampled 
schools 

Participating originally Eligible originally 

sampled schools sampled schools 

(n=145) (n=208) 

Standard Standard Relative Chi-square test 



School characteristic 


Percent 


error 


Percent 


error 


Bias 


bias 


p value 


Region 

Northeast 


16.9 


1.73 


17.5 


0.20 


-0.6 


-3.4 


.750 


Midwest 


23.5 


1.85 


21.7 


0.29 


1.8 


8.3 




South 


36.6 


2.26 


36.2 


0.41 


0.4 


1.1 




West 


23.1 


2.08 


24.6 


0.36 


-1.5 


-6.1 





NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 



As the tables show, the Chi-square tests of independence indicate that there is no evidence of significant 
differences between participating originally sampled schools and eligible originally sampled schools by 
school control, community type, or Census region at the p < .05 percent level. 

The second group of analyses consisted of / tests of the differences between groups on continuous 
variables. Table H-4 shows the means and standard errors for participating originally sampled schools 
and eligible originally sampled schools on the continuous variables, as well as the bias, relative bias, and p 
value for the / test comparisons. 
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Table H.4. Comparisons of participating originally sampled schools and eligible originally sampled schools 

on continuous variables 

Participating originally Eligible originally 

sampled schools sampled schools 



Variable 


(n=145) 

Estimate 


Standard 

error 


(n=208) 

Estimate 


Standard 

error 


Bias 


Relative 

bias 


ttest 
p value 


Total students in school 


1508.8 


80.74 


1537.0 


63.08 


-28.2 


-1.8 


.491 


Age-eligible students in school 


383.8 


21.80 


394.6 


17.23 


-10.8 


-2.7 


.353 


Percentage White students 


62.3 


2.55 


62.3 


2.02 


# 


# 


t 


Percentage Black students 


14.7 


1.63 


14.6 


1.37 


0.1 


0.7 


.902 


Percentage Hispanic students 


17.4 


1.96 


17.2 


1.52 


0.2 


1.1 


.823 


Percentage Asian/Pacific 


Islander students 


4.2 


0.57 


4.5 


0.46 


-0.3 


-6.6 


.383 


Percentage American 
Indian/Alaska Native 


students 


1.4 


0.68 


1.4 


0.51 


# 


# 


t 


Percentage free or reduced- 
price lunch students 


36.0 

(n=135) 


2.11 


33.6 

(n=192) 


1.67 


2.4 


7.1 


,017* 


Percentage male students 


52.2 

(n=138) 


0.71 


52.3 

(n=198) 


0.59 


-0.1 


-0.2 


.787 



t Notapplicable. 

# Rounds to zero. 

*p <.05. Significantly different atthe .05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 



As the table shows, only one of the nine variables examined had a statistically significant difference 
between participating originally sampled schools and eligible originally sampled schools at the p < .05 
level. Participating originally sampled schools had a higher mean percentage of students eligible for free 
or reduced-price lunch than did eligible originally sampled schools. 



The third set of analyses involved logistic regressions predicting school participation in PISA. Two 
regressions were performed, with the dependent variable coded as 1 for participating originally sampled 
schools and as 0 for non-participating eligible originally sampled schools. Regression 1: (1) included all 
public and private schools; (2) excluded the variable for the percentage of students eligible for free or 
reduced-price student lunch (as these data were not available for private schools); (3) excluded the 
variable for the percentage of male students (as these data were missing for 10 of 16 private schools); (4) 
used the percentage of Black, non- Hispanic students as the reference variable for other ethnic 
percentages; and (5) used the rural area category as the reference group on the community type variable, 
and Census region 4 (W est) as the reference group for the Census region variable. Regression 2: (1) 
included only public schools (as the percentage of students at the school eligible for free or reduced- 
price lunch was only available for public schools and the percentage of male students was missing for 
most private schools); (2) excluded the school control variable (public/private); (3) used the percentage 
of White, non- Hispanic students as the reference group for the other ethnic percentages (so that the 
independent effects for Black, non-Hispanic students could be examined); and (4) used same reference 
groups as Regression 1 for the community type and Census region variables. The results of these two 
regressions are shown in Tables H.5 and H.6. 
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Table H.5. Logistic regression 1 for participating originally sampled schools and eligible originally sampled 
schools: parameter estimates 




Estimate 




F test 


Parameter 


(n=208) 


Standard error 


p value 


Intercept 


-0.37 


1.312 


.779 


Total students in school 


# 


t 


t 


Age-eligible students in school 


# 


t 


t 


Central city 


-0.11 


0.463 


.815 


Urban area 


-0.03 


0.507 


.948 


Town 


-0.30 


0.632 


.632 


Northeast 


-0.14 


0.556 


.805 


Midwest 


-0.47 


0.562 


.402 


South 


-0.22 


0.516 


.664 


Percentage White students 


# 


t 


t 


Percentage Hispanic students 


-0.01 


0.012 


.635 


Percentage Asian/Pacific Islander students 


0.01 


0.028 


.655 


Percentage Amer, Indian/Alaska Native students 


# 


t 


t 


Public school 


-0.37 


0.695 


.594 



t Not applicable. 

# Rounds to zero. 

NOTE: The measure of overall fit for the regression model was not statistically significant at the p <.05 level (F = 0.303, p = .991) 
SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009 



Table H.6. Logistic regression 2 for participating originally sampled schools and eligible originally sampled 

schools: parameter estimates 

Estimate F test 



Parameter 


(n=192) 


Standard error 


p value 


Intercept 


-0.96 


1.969 


.626 


Total students in school 


# 


t 


t 


Age-eligible students in school 


0.01 


0.003 


.258 


Central city 


-0.10 


0.555 


.851 


Urban area 


0.11 


0.552 


.838 


Town 


-0.18 


0.670 


.792 


Northeast 


-0.38 


0.617 


.534 


Midwest 


-0.46 


0.626 


.460 


South 


-0.15 


0.545 


.788 


Percentage Black students 


0.01 


0.012 


.381 


Percentage Hispanic students 


0.01 


0.011 


.340 


Percentage Asian/Pacific Islander students 


0.02 


0.031 


.540 


Percentage Amer.lndian/Alaska Native students 


0.02 


0.119 


.855 


Percentage free or reduced-price lunch students 


-0.03 


0.011 


,012* 


Percentage male students 


0.02 


0.034 


.639 



t Notapplicable. 

# Rounds to zero. 

*p <.05. Significantly different at the ,05 level of statistical significance, 

NOTE: The measure of overall fit for the regression model was not statistically significant at the p <.05 level (F =0.713, p = .760). This analysis includes only 
public schools. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 



The results of the regressions were similar to those of the bivariate analyses. The only variable that 
reached statistical significance at the p < .05 level was the percentage of students in the school who were 
eligible for free or reduced-price lunch (analysis was for public schools only). The measures of overall fit 
for the models were not statistically significant. 
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H.5 School Nonresponse Bias Among All Participating Schools — 
Nonadjusted Weights 



The second set of analyses as described in Section H3 were performed to compare all participating 
schools (145 original and 20 replacement) (n=165) with all sampled eligible schools (n=208), using 
nonadjusted weights. Tables H.7 through H.9 show the results for the variables school control 
(public/private), community type (central city, urban area, town, rural area), and Census region 
(Northeast, Midwest, South, West). 

Table H.7. School control for participating schools and sampled eligible schools- nonadjusted weights 





Participating schools 


Sampled eligible schools 








(n=165) 




(n=208) 










Standard 




Standard 




Relative Chi-square test 


School characteristic 


Percent 


error 


Percent 


error 


Bias 


bias p value 


School control 












.571 


Public 


93.0 


1.06 


92.5 


0.41 


0.5 


0.5 


P rivate 


7.0 


1.06 


7.5 


0.41 


-0.5 


-6.7 


NOTE: Detail may notsum to totals due to rounding. 












SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 


Table H.8. Community type for participating schools and sampled eligible schools 


-nonadjusted weights 




Participating schools 


Sampled eligible schools 








(n=165) 




(n=208) 










Standard 




Standard 




Relative Chi-square test 


School characteristic 


Percent 


error 


Percent 


error 


Bias 


bias p value 


Community type 












.236 


Central city 


30.5 


3.63 


30.2 


3.22 


.0.3 


1.0 


Urban area 


32.6 


3.74 


34.9 


3.38 


-2.3 


-6.6 


Town 


14.7 


2.78 


12.7 


2.33 


2.0 


15.7 


Rural area 


22.3 


3.26 


22.1 


2.88 


0.2 


0.9 


NOTE: Detail may notsum to totals due to rounding. 



SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 



Table H.9. Census region for participating schools and sampled eligible schools- nonadjusted weights 
Participating schools Sampled eligible schools 
(n=165) (n=208) 



School characteristic 


Percent 


Standard 

error 


Percent 


Standard 

error 


Bias 


Relative 

bias 


Chi-square test 
p value 


Region 

Northeast 


16.6 


1.33 


17.4 


0.19 


-0.8 


-4.6 


.197 


Midwest 


24.3 


1.28 


21.7 


0.28 


2.6 


12.0 




South 


36.3 


1.73 


36.0 


0.39 


0.3 


0.8 




West 


22.8 


1.61 


24.9 


0.25 


-2.1 


-8.4 





NOTE: Detail may notsum to totals due to rounding, 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 



As the tables show, the Chi-square tests of independence indicate that there is no evidence of significant 
differences between participating schools and sampled eligible schools by school control, community 
type, or Census region at the p < .05 percent level. However, the absolute values of relative bias for 
schools in towns and in the Midwest region are greater than 1 0 percent, which indicates potential bias 
even though no statistically significant differences were detected. 

Table H.10 shows the results of comparisons on the continuous variables. 
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Table H.10. Comparisons of participating schools and sampled eligible schools on continuous variables- 

nonadjusted weights 

Participating schools Sampled eligible schools 
(n=165) (n=208) 



Variable 


Estimate 


Standard 

error 


Estimate 


Standard 

error 


Bias 


Relative 

bias 


ttest 
p value 


Total students in school 


1500.1 


74.97 


1536.8 


63.79 


-36.7 


-2.4 


.233 


Age-eligible students in school 


380.9 


20.24 


392.6 


17.38 


-11.7 


-3.0 


.187 


Percentage White students 


62.3 


2.37 


62.7 


1.99 


-0.4 


-0.6 


.679 


Percentage Black students 


14.4 


1.49 


13.7 


1.25 


0.7 


5.1 


.208 


Percentage Hispanic students 


17.8 


1.83 


17.7 


1.52 


0.1 


0.6 


.897 


Percentage Asian/Pacific 


Islander students 


4.3 


0.57 


4.6 


0.50 


-0.3 


-6.5 


.242 


Percentage American 
Indian/Alaska Native 


students 


1.2 


0.60 


1.4 


0.50 


-0.2 


-14.3 


.447 


Percentage free or reduced- 
price lunch students 


35.3 

(n=154) 


1.93 


33.8 

(n=192) 


1.65 


1.5 


4.4 


.024* 


Percentage male students 


52.1 

(n=157) 


0.62 


52.3 

(n=198) 


0.58 


-0.2 


0.4 


.622 



*p <.05. Significantly different atthe .05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 



As for the comparisons of originally sampled schools, only one of the nine variables examined had 
statistically significant differences between participating schools and sampled eligible schools at the p < 
.05 level. Participating schools had a higher mean percentage of students eligible for free or reduced- 
price lunch than did sampled eligible schools. However, the absolute value of relative bias for percentage of 
American Indian/ Alaska native students is greater than 10 percent, which indicates potential bias even 
though no statistically significant differences were detected. 



Two regressions were performed, with the dependent variable coded as 1 for participating schools and as 
0 for non-participating sampled eligible schools. Regression 1: (1) included all public and private 
schools; (2) excluded the variable for the percentage of students eligible for free or reduced-price student 
lunch (as these data were not available for private schools); (3) excluded the variable for the percentage 
of male students (as these data were missing for 10 of 16 private schools); (4) used the percentage of 
Black, non-Hispanic students as the reference variable for other ethnic percentages; and (5) used the 
rural area category as the reference group on the community type variable, and Census region 4 (West) as 
the reference group for the Census region variable. Regression 2: (1) included only public schools (as 
the percentage of students at the school eligible for free or reduced-price lunch were only available for 
public schools and the percentage of male students was missing for most private schools); (2) excluded 
the school control variable (public /private); (3) used the percentage of White, non-Hispanic students as 
the reference group for the other ethnic percentages (so that the independent effects for Black, non- 
Hispanic students could be examined); and (4) used same reference groups as Regression 1 for the 
community type and Census region variables. Tables H.ll and H.12 show the results of the two 
regressions. 
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Table H.ll. Logistic regression 1 for participating schools and sampled eligible schools on continuous 
variables- nonadjusted weights: parameter estimates 




Estimate 




F test 


Parameter 


(n=208) 


Standard error 


p value 


Intercept 


-2.78 


2.058 


.178 


Total students in school 


-0.00 


0.001 


.473 


Age-eligible students in school 


0.00 


0.004 


.721 


Central city 


-0.21 


0.557 


.752 


Urban area 


-0.18 


0.567 


.767 


Town 


-1.64 


1.318 


.214 


Northeast 


-0.28 


0.613 


.653 


Midwest 


-1.24 


0.741 


.097 


South 


-0.23 


0.581 


.699 


Percentage White students 


0.02 


0.019 


.380 


Percentage Hispanic students 


0.00 


0.020 


.881 


Percentage Asian/Pacific Islander students 


0.03 


0.033 


.447 


Percentage American Indian/Alaska Native students 


0.04 


0.179 


.840 


Public school 


0.38 


0.678 


.577 


NOTE: The measure of overall fit for the regression model was not statistically significant at the p <.05 level (F 


= 0.836, p =,622) 




SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 




Table H.12. Logistic regression 2 for participating schools and sampled eligible schools on continuous 




variables- nonadjusted weights: 


parameter estimates 








Estimate 




F test 


Parameter 


(n=192) 


Standard error 


p value 


Intercept 


-0.49 


2.561 


.849 


Total students in school 


-0.00 


0.001 


.302 


Age-eligible students in school 


0.01 


0.005 


.284 


Central city 


0.00 


0.680 


.994 


Urban area 


-0.22 


0.679 


.743 


Town 


-1.54 


1.364 


.259 


Northeast 


-0.67 


0.721 


.355 


Midwest 


-1.33 


0.848 


.118 


South 


-0.25 


0.602 


.683 


Percentage Black students 


-0.00 


0.018 


.806 


Percentage Hispanic students 


0.00 


0.012 


.836 


Percentage Asian/Pacific Islander students 


0.03 


0.035 


.402 


Percentage American Indian/Alaska Native students 


0.04 


0.180 


.846 


Percentage free or reduced-price lunch students 


-0.02 


0.012 


.057 


Percentage male students 


0.01 


0.045 


.892 



NOTE: The measure of overall fit for the regression model was not statistically significant at the p < 05 level (F =0.980, p= .475). This analysis includes only 
public schools. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 



The results of the regressions were similar to those for the original sample. The overall measures of fit 
were not statistically significant at the p < .05 level. However, with the inclusion of all participating 
schools, none of the estimates for individual variables (including the percentage of students eligible for 
free or reduced-price lunch) were statistically significant. 

H.6 School Nonresponse Bias — Nonresponse Adjusted Weights 

The same analyses as described above comparing all participating schools (original and replacement) and 
all eligible sampled schools were repeated using the PISA nonresponse adjusted weights. These weights 
were calculated based on response rates within sampling strata, not based on nonresponse bias analyses 
or poststratification. Tables H.13 through H.15 show the results for the variables school control 
(public/private), community type (central city, urban area, town, rural area), and Census region 
(Northeast, Midwest, South, West). 
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Table H.13. School control for participating schools and sampled eligible schools-adjusted weights 
Participating schools Sampled eligible schools 
(n=165) (n=208) 

Standard Standard Relative Chi-square test 

School characteristic Percent error Percent error Bias bias p value 

School control ,663 

Public 92.5 1,26 92.2 0.62 0.3 0.3 

Private 725 L26 U 062 A3 A8 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 



Table H.14. Community type for participating schools and sampled eligible schools-adjusted weights 
Participating schools Sampled eligible schools 
(n=165) (n =208) 



School characteristic 


Percent 


Standard 

error 


Percent 


Standard 

error 


Bias 


Relative 

bias 


Chi-square test 
p value 


Community type 
Central city 


30.2 


3.63 


30.0 


3.23 


.0.2 


0.7 


.320 


Urban area 


34.5 


3.87 


36.1 


3.47 


-1.6 


-4.4 




Town 


14.2 


2.72 


12.6 


2.34 


1.6 


12.7 




Rural area 


21.1 


3.13 


21.2 


2.81 


-0.1 


0.5 





NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 



Table H.15. Census region for participating schools and sampled eligible schools-adjusted weights 
Participating schools Sampled eligible schools 
(n=165) (n=208) 



School characteristic 


Percent 


Standard 

error 


Percent 


Standard 

error 


Bias 


Relative 

bias 


Chi-square test 
p value 


Region 

Northeast 


16.5 


1.40 


17.2 


0.41 


-0.7 


-4.1 


.351 


Midwest 


21.8 


1.27 


20.1 


0.47 


1.7 


8.5 




South 


36.8 


1.84 


36.4 


0.65 


0.4 


1.1 




West 


24.9 


1.79 


26.3 


0.55 


-1.4 


-5.3 





NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 



As the tables show, the results were very similar to those using non-adjusted weights. The Chi-square 
tests of independence indicate that there is no evidence of significant differences between participating 
schools and sampled eligible schools by school control, community type, or Census region at the p < .05 
percent level. However, the absolute value of relative bias for schools in towns is greater than 10 
percent, which indicates potential bias even though no statistically significant differences were detected. 



Table H.16 shows the results for the continuous variables. 
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Table H.16. Comparisons of participating schools and sampled eligible schools on continuous variables- 

adjusted weights 

Participating schools Sampled eligible schools 
(n =165) (n =208) 



Variable 


Estimate 


Standard 

error 


Estimate 


Standard 

error 


Bias 


Relative 

bias 


ttest 
p value 


Total students in school 


1523.7 


77.00 


1550.3 


66.06 


-26.6 


-1.7 


.305 


Age-eligible students in school 


387.4 


20.83 


396.1 


17.94 


-8.7 


-2.2 


.247 


Percentage White students 


61.9 


2.37 


62.3 


2.02 


-0.4 


-0.6 


.622 


Percentage Black students 


14.3 


1.46 


13.7 


1.25 


0.6 


4.4 


.222 


Percentage Hispanic students 


18.1 


1.86 


18.0 


1.57 


0.1 


0.6 


.813 


Percentage 

Asian/Pacificlslander 


students 


4.4 


0.58 


4.6 


0.51 


-0.2 


-4.3 


.276 


Percentage American 
Indian/Alaska Native 


students 


1.3 


0.61 


1.4 


0.53 


-0.1 


-7.1 


.466 


Percentage free or reduced- 
price lunch students 


35.4 

(n=154) 


1.95 


34.1 

(n=192) 


1.70 


1.3 


3.8 


.022* 


Percentage male students 


52.3 

(n=157) 


0.68 


52.4 

(n=198) 


0.61 


-0.1 


-0.2 


.713 



*p <.05. Significantly different at the .05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 

As for the nonadjusted weight analyses, only one of the nine variables examined had a statistically 
significant difference between participating schools and sampled eligible schools at the p < .05 level. 
Participating schools had a higher mean percentage of students eligible for free or reduced-price lunch 
than did sampled eligible schools. 

Two regressions were performed using the adjusted weights, with the dependent variable coded as 1 for 
participating schools and as 0 for non-participating sampled eligible schools. Regression 1: (1) included 
all public and private schools; (2) excluded the variable for the percentage of students eligible for free or 
reduced-price student lunch (as these data were not available for private schools); (3) excluded the 
variable for the percentage of male students (as these data were missing for 10 of 16 private schools); (4) 
used the percentage of Black, non- Hispanic students as the reference variable for other ethnic 
percentages; and (5) used the rural area category as the reference group on the community type variable, 
and Census region 4 (West) as the reference group for the Census region variable. Regression 2: (1) 
included only public schools (as the percentage of students at the school eligible for free or reduced- 
price lunch were only available for public schools and the percentage of male students was missing for 
most private schools); (2) excluded the school control variable (public/private); (3) used the percentage 
of White, non-Hispanic students as the reference group for the other ethnic percentages (so that the 
independent effects for Black, non-Hispanic students could be examined); and (4) used same reference 
groups as Regression 1 for the community type and Census region variables. The results of these two 
regressions are shown in Tables H.17 and H.18. 
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Table H.17. Logistic regression 1 for participating schools and sampled eligible schools-adjusted weights: 



parameter estimates 



Parameter 


Estimate 

(n=208) 


Standard error 


F test 
p value 


Intercept 


-3.06 


1.997 


.127 


Total students in school 


-0.00 


0.001 


.517 


Age-eligible students in school 


0.00 


0.005 


.413 


Central city 


-0.15 


0.583 


.803 


Urban area 


-0.05 


0.548 


.927 


Town 


-1.52 


1.356 


.264 


Northeast 


-0.13 


0.606 


.828 


Midwest 


-1.00 


0.734 


.173 


South 


-0.13 


0.582 


.820 


Percentage White students 


0.02 


0.019 


.394 


Percentage Hispanic students 


0.00 


0.020 


.855 


Percentage Asian/Pacific Islander students 


0.03 


0.033 


.426 


Percentage American Indian/Alaska Native students 


0.04 


0.177 


.841 


Public school 


0.33 


0.685 


.627 



NOTE: The measure of overall fit for the regression model was not statistically significant at the p <.05 level (F = 0.836, p = ,622) 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 

Table H.18. Logistic regression 2 for participating schools and sampled eligible schools-adjusted weights: 



parameter estimates 



Parameter 


Estimate 

(n=192) 


Standard error 


F test 
p value 


Intercept 


-0.73 


2.578 


.776 


Total students in school 


-0.00 


0.001 


.317 


Age-eligible students in school 


0.01 


0.005 


.304 


Central city 


-0.14 


0.691 


.836 


Urban area 


0.16 


0.589 


.790 


Town 


-1.44 


1.381 


.299 


Northeast 


-0.51 


0.712 


.475 


Midwest 


-1.07 


0.842 


.206 


South 


-0.11 


0.597 


.850 


Percentage Black students 


-0.00 


0.017 


.810 


Percentage Hispanic students 


0.00 


0.012 


.770 


Percentage Asian/Pacific Islander students 


0.03 


0.036 


.444 


Percentage American Indian/Alaska Native students 


0.04 


0.176 


.841 


Percentage free or reduced-price lunch students 


-0.02 


0.012 


.065 


Percentage male students 


0.00 


0.045 


.957 



NOTE: The measure of overall fit for the regression model was not statistically significant at the p <.05 level (F =0.816, p = .651). This analysis includes only 
public schools. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 



The results of the regressions were very similar to those using nonadjusted weights. The overall 
measures of fit were not statistically significant at the p < .05 level, and none of the estimates for 
individual variables were statistically significant. 

H.7 Methodology for Analysis of Item Nonresponse Bias 

When response rates on any of the items on the School Questionnaire or Student Questionnaire were 
less than 85 percent, those who responded on specific items were compared to those who did not 
respond. These analyses used final nonresponse-adjusted weights. 

For items on the School Questionnaire, respondents to an item were compared with non-respondents 
using data from the sampling frame. The specific variables on which these two groups were compared 
include both categorical and continuous variables. The categorical variables were: 
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• School control (public/private); 

• Community type (central city, urban area, town, rural area); and 

• Region (Northeast, Midwest, South, West). 

The continuous variables were: 

Total number of students in the school; 

Estimated number of age-eligible students in the school; 

Percentage of students eligible for free or reduced-price student lunch. 

Percentage of White, non- Hispanic students; 

Percentage of Black, non-Hispanic students; 

Percentage of Hispanic students; 

Percentage of Asian/Pacific Islander non-Hispanic students; 

Percentage of American Indian/Alaska Native non-Hispanic students; and 
Percentage of male students. 

Both student and school-level variables were used for analyses of items with missing data on the Student 
Questionnaire. School-level data came from the PISA sampling frame, while student-level data came 
from the Student Questionnaire. If data were missing on the comparison items on the Student 
Questionnaire, those cases were excluded from the analysis. 

The specific variables on which these two groups were compared included both categorical and 
continuous variables. The categorical variables were: 

Student gender; 

Student grade level (9 or lower, 10 or higher) 

Hispanic ethnicity (yes/no); 

White race (yes/no); 

Black race (yes/no); 

Asian race (yes/no); 

American Indian or Alaska Native race (yes/ no); 

Native Hawaiian or other Pacific Islander race (yes/no); 

School control (public/private); 

Community type (central city, urban area, town, rural area); and 
Region (Northeast, Midwest, South, West). 

The continuous variables were: 



• Total number of students in the school; 

• Estimated number of age-eligible students in the school; and 

• Percentage of students eligible for free or reduced-price student lunch. 

For categorical variables, the chi-square statistic was used to test differences between the item 
respondents and non-respondents. The percentages and standard errors for the two groups, the 
difference in percentage (bias), and chi-square p value are presented in tables. 
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For continuous variables, summary means for the two groups were calculated and the differences in 
means were tested using / tests. The means and standard errors for the two groups, the difference in 
means (bias), and p value are presented in tables. 

Because of the stradfied sampling design used for PISA, all analyses were done using Wesvar in order to 
generate error terms. 

H.8 Item Nonresponse Bias — School Questionnaire 

Response rate was less than 85 percent on three items on the School Questionnaire. These items were: 



How many of the following teachers are on the staff of your school — part-time teachers in total? (Q9a2) 

How many of the following teachers are on the staff of your school — part-time teachers fully certified by 
the state in the main assignment field? (Q9b2) 

How many of the following teachers are on the staff of your school — part-time teachers with at least a 
bachelor’s degree? (Q9c2) 



A review of School Questionnaire data suggests that at least some respondents may have skipped these 
items because they did not have any part-time teachers (though they were asked to enter “0”). The 
results of item nonresponse analyses (summarized in Table H-19) indicate that respondents on items 
Q9a2 and Q9c2 differed from nonrespondents on at least two key variables. On item Q9a2, 
respondents more often than nonrespondents reported coming from private schools (34 percent versus 
5 percent) and from schools with higher percentages of White students (averages equal 76 percent versus 
52 percent). On item Q9c2, respondents more often than nonrespondents reported coming from 
private schools (44 percent versus 3 percent), from schools with higher percentages of White students 
(averages equal 79 percent versus 60 percent), and from schools with lower percentages of Black 
students (averages equal 6 percent versus 25 percent). The details of these analyses are presented in data 
tables in Section Fll. 

Table H.19. Summary of statistically significant differences between respondents and nonrespondents on 
specific items on the school questionnaire- p values 







Item 




Comparison variable 


Q9a2 


Q9b2 


Q9c2 


School control 
Community type 
Region 

Total students in school 
Age-eligible students in school 


p=.027* 




p=.001* 


Percentage White students 
Percentage Black students 
Percentage Hispanic students 
Percentage Asian/Pacific Islander students 
Percentage American Indian/Alaska Native students 
Percentage free or reduced-price lunch students 
Percentage male students 


p=.041* 




p=.029* 

p=.026* 



*p <.05. Significantly different atthe .05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 
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H.9 Item Nonresponse Bias — Student Questionnaire 

There were 10 items on the Student Questionnaire that had less than an 85 percent response rate: 

• Who usually lives at home with you - sister(s) (including stepsisters) (Q12d); 

• Who usually lives at home with you — grandparent (s) (Q12e); 

• Who usually lives at home with you — others (e.g., cousin) (Q12f); 

• Does your mother have any of the following degrees, certificates, or diplomas — Master’s, 

doctoral, or professional degree such as medicine or law (Q15a) 

• Does your mother have any of the following degrees, certificates, or diplomas — Associate’s 
degree (2-year degree from a community college) (Q15c); 

• Does your mother have any of the following degrees, certificates, or diplomas — vocational or 
technical certificate/ diploma after high school (such as cosmetology or auto mechanics) (Q15d); 

• Does your father have any of the following degrees, certificates, or diplomas — Master’s, 
doctoral, or professional degree such as medicine or law (Q18a); 

• Does your father have any of the following degrees, certificates, or diplomas — Bachelor’s degree 
(4-year college degree) (Q18b); 

• Does your father have any of the following degrees, certificates, or diplomas — Associate’s degree 
(2-year degree from a community college) (Q18c); and 

• Does your father have any of the following degrees, certificates, or diplomas — vocational or 
technical certificate/diploma after high school (such as cosmetology or auto mechanics) (Q18d). 

The nonresponse bias results on these ten items are summarized in Tables H.20. On all ten items there 
were significant differences between respondents and nonrespondents on from 2 to 10 of the examined 
comparison variables. In general, respondents to these items more often than nonrespondents reported 
being female, being in grade 10 or higher, being White, Hispanic, or American Indian/Alaska Native, 
living in rural areas or the West region, and attending smaller schools and schools with lower percentages 
of students receiving free or reduced-price lunch. Respondents less often than nonrespondents reported 
being Black/African American and living in center cities or the Northeast region. The details of these 
analyses are presented in data tables in Section Fll. 
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Table H.20. Summary of statistically significant differences between respondents and nonrespondents on 

specific items on the student questionnaire- p values 

Item 



Comparison variable 


Q12d 


Q12e 


Q12f 


Q 15a 


Q 15c 


Q15d 


Q 18a 


Q 18b 


Q 18c 


Q18d 


Student gender 








p=.001* 


p=.000* 


p=.000* 


p=.013* 


p=.023* 


p=.000* 


p=.001* 


Student grade level 








p=,017* 


p=,005* 


p=.035* 


p=.005* 


p=.000* 


p=.003* 


p=.018* 


Hispanic or Latino 










p=.004* 


p=.001* 








p=,006* 


White 


p=,004* 




p=.015* 


p=.006* 


p=.003* 


p=.020* 


p=.000* 


p=.000* 


p=.000* 


p=.010* 


Black or African American 


p=.000* 




p=.007* 


p=.001* 


p=,000* 


p=.000* 


p=,000* 


p=,000* 


p=.000* 


p=.000* 


Asian 


p=.000* 


p=.000* 


p=.015* 


p=.008* 


p=.003* 


p=.004* 






p=.009* 


p=.013* 


American Indian or Alaska Native 
Native Hawaiian or Other Pacific 








p=.000* 


p=.000* 


p=.000* 


p=.000* 


p=.000* 


p=.009* 


p=.000* 


Islander 
School control 
Community type 










p=,036* 






p=.010* 


p=.000* 


p=,000* 


Region 


p=,004* 


p=.011* 


p=.009* 


p=.002* 


p=.006* 


p=.001* 


p=.000* 


p=.004* 


p=.002* 


p=.000* 


T otal number of students in school 
Estimated number of age-eligible 














p=,045* 




p=.009* 




students in school 
Percentage free or reduced-price 


















p=.024* 




lunch students 












p=.013* 




p=,002* 







*p <.05. Significantly different at the ,05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 



H.10 Discussion 



In examining school-level nonresponse, the bivariate analysis results showed that only one of the 12 
variables examined (percentage of students eligible for free or reduced-price lunch) had a statistically 
significant relationship with school participation. This result occurred for schools in the original sample 
and for all participating schools both before and after nonresponse adjustments were applied. In logistic 
regression analyses, this same variable was statistically significant for original schools but did not reach 
statistical significance for the two analyses of all participating schools. Overall measures of fit were not 
statistically significant in any of the logistic regression analyses. Thus, the overall regression equations 
did not provide statistically significant evidence of differences between school-level respondents and 
nonrespondents. 

The results on item level nonresponse indicated that three items on the School Questionnaire and 1 0 
items on the Student Questionnaire had less than 85 percent response rates. The results of nonresponse 
analyses suggest that respondents on 12 of these 13 items were different from nonrespondents. On the 
School Questionnaire, respondents more often than nonrespondents reported coming from private 
schools and from schools with higher percentages of White students. On the Student Questionnaire, 
respondents more often than nonrespondents reported being female, being in grade 10 or higher, being 
White, Hispanic, or American Indian/Alaska Native, living in rural areas or the West region, and 
attending smaller schools and schools with lower percentages of students receiving free or reduced-price 
lunch. Respondents less often than nonrespondents reported being Black/African American and living 
in center cities or the Northeast region. 
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H.11 Additional Item Nonresponse Data Tables 

Table H.21. Types of schools for respondents and nonrespondents to school questionnaire item Q9a2 



(number of part-time teachers) 



- 


Respondents 

(n=120) 

Standard 
Percent error 


Nonrespondents 

(n=45) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


School control 

Public 


65.8 


10.5 


94.6 


3.90 


-28.8 


-43.77 


.027* 


Private 


34.2 


10.5 


5.4 


3.90 


28.8 


84.21 




Community type 

Central city 


29.3 


12.26 


18.6 


7.52 


10.7 


36.52 


.712 


Urban area 


30.5 


12.10 


25.7 


8.46 


4.8 


15.74 




Town 


9.0 


3.20 


18.9 


9.86 


-9.9 


-110.00 




Rural area 


31.3 


9.51 


36.8 


13.10 


-5.5 


-17.57 




Region 

Northeast 


15.2 


8.28 


17.4 


6.79 


-2.2 


-14.47 


.081 


Midwest 


13.8 


3.82 


46.8 


11.65 


-33.0 


-239.13 




South 


46.0 


10.23 


22.5 


7.66 


23.5 


51.09 




West 


25.0 


8.77 


13.3 


8.83 


11.7 


46.80 





*p <.05. Significantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 



Table H.22. Comparisons of participating and eligible sampled schools to school questionnaire item Q9a2 
(number of part-time teachers) 



Respondents Nonrespondents 

(n=120) (n=45) 



Variable 


Estimate 


Standard 

error 


Estimate 


Standard 

error 


Bias 


Relative 

bias 


Chi-square 
p value 


Total students in school 


620.6 


86.02 


776.7 


144.96 


-156.1 


-25.15 


.362 


Age-eligible students in school 


113.5 


22.49 


196.5 


37.32 


-83.0 


-73.13 


.061 


Percentage White students 


76.0 


4.86 


52.0 


10.58 


24.0 


31.58 


.041* 


Percentage Black students 


10.1 


4.24 


25.7 


9.41 


-15.6 


-154.46 


.138 


Percentage Hispanic students 


11.3 


2.73 


10.5 


3.00 


0.8 


7.08 


.837 


Percentage Asian/Pacific 


Islander students 


2.0 


0.47 


2.5 


0.88 


-0.5 


-25.00 


.574 


Percentage American 
Indian/Alaska Native 


students 


0.6 


0.15 


9.3 


8.99 


-8.7 


-1450.00 


.332 


Percentage free or reduced- 
price lunch students 


41.4 

(n=lll) 


6.86 


37.7 

(n=43) 


8.15 


3.7 


8.94 


.729 


Percentage male students 


53.1 

(n=112) 


1.73 


49.6 

(n=45) 


5.61 


3.5 


6.6 


.551 



*p <.05. Significantly different at the .05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 
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Table H.23. Types of schools for respondents and nonrespondents to school questionnaire item Q9b2 



(number of fully certified part-time teachers) 



- 


Respondents 

(n=94) 

Standard 
Percent error 


Nonrespondents 

(n=71) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


School control 

Public 


67.8 


12.01 


75.3 


14.61 


-7.5 


-11.06 


.695 


Private 


32.2 


12.01 


24.8 


14.61 


7.4 


22.98 




Community type 

Central city 


26.0 


13.86 


29.4 


17.18 


-3.4 


-13.08 


.818 


Urban area 


26.0 


13.99 


34.8 


14.51 


-8.8 


-33.85 




Town 


9.2 


3.84 


12.9 


6.06 


-3.7 


-40.22 




Rural area 


38.8 


11.73 


22.9 


9.76 


15.9 


40.98 




Region 

Northeast 


8.1 


2.30 


26.3% 


14.45 


-18.2 


-224.69 


.380 


Midwest 


16.5 


5.05 


24.0% 


9.03 


-7.5 


-45.45 




South 


46.4 


11.29 


35.3% 


16.21 


11.1 


23.92 




West 


29.0 


11.50 


14.4% 


6.88 


14.6 


50.34 





NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 



Table H.24. Comparisons of respondents and nonrespondents to school questionnaire item Q9b2 (number of 

fully certified part-time teachers) 

Respondents Nonrespondents 

(n=94) (n=71) 



Variable 


Estimate 


Standard 

error 


Estimate 


Standard 

error 


Bias 


Relative 

bias 


ttest 
p value 


Total students in school 


611.5 


95.06 


700.3 


130.70 


-88.8 


-14.52 


.582 


Age-eligible students in school 


124.8 


25.92 


132.9 


40.19 


-8.1 


-6.49 


.866 


Percentage White students 


76.4 


3.87 


65.2 


9.90 


11.2 


14.66 


.301 


Percentage Black students 


6.7 


1.57 


21.5 


8.28 


-14.8 


-220.90 


.084 


Percentage Hispanic students 


14.0 


3.49 


7.2 


2.01 


6.8 


48.57 


.100 


Percentage Asian/Pacific 
Islander students 


2.2 


0.62 


2.0 


0.80 


0.2 


9.09 


.851 


Percentage American 
Indian/Alaska Native students 


0.7 


0.19 


4.1 


3.96 


-3.4 


-485.71 


.403 


Percentage free or reduced-price 
lunch students 


36.6 

(n=88) 


3.71 


45.6 

(n=66) 


12.29 


-9.0 


-24.59 


.486 


Percentage male students 


54.0 

(n=89) 


2.11 


50.1 

(n=68) 


3.01 


3.9 


7.22 


.282 



SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.25. Types of schools for respondents and nonrespondents to school questionnaire item 



Q9c2 (number of part-time teachers with bachelor's degrees) 



- 


Respondents 

(n=120) 

Standard 
Percent error 


Nonrespondents 

(n=45) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


School control 

Public 


56.4 


11.85 


97.3 


2.06 


-40.9 


-72.52 


.001* 


Private 


43.6 


11.85 


2.7 


2.06 


40.9 


93.81 




Community type 

Central city 


24.0 


12.58 


33.5 


18.82 


-9.5 


-39.58 


.799 


Urban area 


33.9 


14.82 


21.8 


8.22 


12.1 


35.69 




Town 


8.4 


3.52 


15.0 


6.95 


-6.6 


-78.57 




Rural area 


33.7 


11.31 


29.7 


11.63 


4.0 


11.87 




Region 

Northeast 


17.2 


10.35 


12.8 


4.89 


4.4 


25.58 


.408 


Midwest 


12.3 


3.96 


32.9 


11.24 


-20.6 


-167.48 




South 


42.7 


11.09 


40.2 


16.96 


2.5 


5.85 




West 


27.8 


10.79 


14.1 


7.26 


13.7 


49.28 





*p <.05. Significantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 



Table H.26. Comparisons of respondents and nonrespondents to school questionnaire item Q9c2 (number 

of part-time teachers with bachelor's degrees) 

Respondents Nonrespondents 

(n=94) (n=71) 



Variable 


Estimate 


Standard 

error 


Estimate 


Standard 

error 


Bias 


Relative 

bias 


ttest 
p value 


Total students in school 


583.1 


100.81 


767.1 


84.07 


-184.0 


-31.56 


.167 


Age-eligible students in school 


114.6 


24.79 


152.9 


44.62 


-38.3 


-33.42 


.453 


Percentage White students 


78.5 


4.21 


59.5 


7.51 


19.0 


24.20 


.029* 


Percentage Black students 


6.3 


1.56 


24.8 


8.04 


-18.5 


-293.65 


.026* 


Percentage Hispanic students 


12.4 


3.47 


8.9 


1.61 


3.5 


28.23 


.342 


Percentage Asian/Pacific 


Islander students 


2.2 


0.59 


2.0 


0.56 


0.2 


9.09 


.804 


Percentage American 
Indian/Alaska Native 


students 


0.6 


0.18 


4.8 


4.61 


-4.2 


-700.00 


.373 


Percentage free or reduced- 
price lunch students 


36.9 

(n=85) 


4.02 


44.4 

(n=69) 


11.10 


-7.5 


-20.33 


.532 


Percentage male students 


54.5 

(n=86) 


2.18 


49.9 

(n=71) 


2.72 


4.6 


8.44 


.191 



*p <.05. Significantly different at the .05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 
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Table H.27. Types of students for respondents and nonrespondents to student questionnaire 
item Q12d (at home- sisters) 



Respondents 

(n=4367) 

Standard 
Estimate error 


Nonrespondents 

(n=866) 

Standard 
Estimate error 


Bias 


Relative 

bias 


Chi-square 
p value 


Student gender 

Female 


48.9 


0.81 


47.4 


1.83 


1.5 


3.07 


0.436 


Male 


51.1 


0.81 


52.6 


1.83 


-1.5 


-2.94 




Student grade level 

9 or Lower 


10.7 


0.49 


12.9 


1.19 


-2.2 


-20.56 


0.084 


10 or Higher 


89.3 


0.49 


87.1 


1.19 


2.2 


2.46 




Hispanic or Latino (Respondent n=4346, Nonrespondent n =824) 

Yes 23.3 0.65 22.9 


1.53 


0.4 


1.72 


0.829 


No 


76.7 


0.65 


77.1 


1.53 


-0.4 


-0.52 




White (Nonrespondent n =850) 

Yes 


67.9 


0.72 


62.4 


1.77 


5.5 


8.10 


0.004* 


No 


32.1 


0.72 


37.6 


1.77 


-5.5 


-17.13 




Black or African American (Nonrespondent n =850) 

Yes 14.5 0.55 


20.2 


1.42 


-5.7 


-39.31 


0.000* 


No 


85.5 


0.55 


79.8 


1.42 


5.7 


6.67 




Asian (Nonrespondent n =850) 

Yes 


5.6 


0.36 


2.9 


0.59 


2.7 


48.21 


0.000* 


No 


94.4 


0.36 


97.1 


0.59 


-2.7 


-2.86 




American Indian or Alaska Native (Nonrespondent n =850) 

Yes 5.5 0.36 4.6 


0.75 


0.9 


16.36 


0.262 


No 


94.5 


0.36 


95.4 


0.75 


-0.9 


-0.95 




Native Hawaiian or Other Pacific Islander (Nonrespondent n =850) 

Yes 2.7 0.25 3.6 


0.66 


-0.9 


-33.33 


0.195 


No 


97.3 


0.25 


96.4 


0.66 


0.9 


0.92 




School control 

Public 


91.3 


0.37 


90.6 


1.35 


0.7 


0.77 


0.667 


Private 


8.7 


0.37 


9.4 


1.35 


-0.7 


-8.05 




Community type 

Central city 


29.0 


0.71 


31.1 


1.72 


-2.1 


-7.24 


0.554 


Urban area 


34.6 


0.77 


32.6 


1.74 


2.0 


5.78 




Town 


14.9 


0.55 


14.1 


1.20 


0.8 


5.37 




Rural area 


21.5 


0.62 


22.2 


1.42 


-0.7 


-3.26 




Region 

Northeast 


15.0 


0.29 


19.3 


1.30 


-4.3 


-28.67 


0.004* 


Midwest 


23.4 


0.31 


19.6 


1.24 


3.8 


16.24 




South 


35.8 


0.41 


37.7 


1.67 


-1.9 


-5.31 




West 


25.8 


0.34 


23.5 


1.42 


2.3 


8.91 





*p <.05. Significantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.28. Comparisons of respondents and nonrespondents to student questionnaire item Q12d (at 



home- sisters) 




Respondents 

(n=4367) 




Nonrespondents 

(n=866) 










Standard 




Standard 




Relative 


ttest 


Variable 


Estimate 


error 


Estimate 


error 


Bias 


bias 


p value 


Total number of students in 
















school 


1457.6 


14.67 


1429.0 


34.52 


28.6 


1.96 


0.456 


Estimated number of age- 
eligible students in school 


369.5 


4.02 


359.0 


9.67 


10.5 


-4.94 


0.329 


Percentage free or reduced- 
price lunch students 


34.4 

(n=4080) 


0.38 


36.1 

(n=808) 


0.92 


-1.7 


3.07 


0.077 



SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.29. Types of Students for respondents and nonrespondents to student questionnaire 



item Q12e (at home- grandparents) 



— 


Respondents 

(n=3769) 

Standard 
Percent error 


Nonrespondents 

(n=1464) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


Student gender 














0.311 


Female 


48.2 


0.87 


49.9 


1.40 


-1.7 


-3.53 




Male 


51.8 


0.87 


50.1 


1.40 


1.7 


3.28 




Student grade level 














0.992 


9 or Lower 


11.1 


0.54 


11.0 


0.84 


0.1 


0.90 




10 or Higher 


89.1 


0.54 


89.0 


0.84 


0.0 


0.00 




Hispanic or Latino (Respondents n=3750, Nonrespondents n=1420) 






0.370 


Yes 


22.9 


0.71 


24.1 


1.18 


-1.2 


-5.24 




No 


77.1 


0.71 


75.9 


1.18 


1.2 


1.56 




White (Nonrespondents n=1448) 












0.055 


Yes 


67.8 


0.79 


64.9 


1.31 


2.9 


4.28 




No 


32.2 


0.79 


35.2 


1.31 


-3.0 


-9.32 




Black or African American (Nonrespondents n=1448) 








0.056 


Yes 


14.8 


0.61 


17.0 


1.00 


-2.2 


-14.86 




No 


85.2 


0.61 


83.0 


1.00 


2.2 


2.58 




Asian (Nonrespondents n=1448) 












0.000* 


Yes 


5.8 


0.39 


3.5 


0.49 


2.3 


39.66 




No 


94.2 


0.39 


96.5 


0.49 


-2.3 


-2.44 




American Indian or Alaska Native (Nonrespondents n=1448) 








0.243 


Yes 


5.6 


0.39 


4.8 


0.58 


0.8 


14.29 




No 


94.4 


0.39 


95.2 


0.58 


-0.8 


-0.85 




Native Hawaiian or Other Pacific Islander (Nonrespondents n=1448) 






0.478 


Yes 


2.8 


0.27 


3.2 


0.47 


-0.4 


-14.29 




No 


97.2 


0.27 


96.9 


0.47 


0.3 


0.31 




School control 














0.668 


Public 


91.3 


0.43 


90.8 


0.99 


0.5 


0.55 




P rivate 


8.7 


0.43 


9.2 


0.99 


-0.5 


-5.75 




Community type 














0.194 


Central city 


29.1 


0.77 


30.0 


1.30 


-0.9 


-3.09 




Urban area 


35.0 


0.82 


32.4 


1.34 


2.6 


7.43 




Town 


14.2 


0.58 


16.2 


0.98 


-2.0 


-14.08 




Rural area 


21.7 


0.67 


21.4 


1.06 


0.3 


1.38 




Region 














0.011* 


Northeast 


14.7 


0.37 


18.4 


0.92 


-3.7 


-25.17 




Midwest 


23.4 


0.40 


21.0 


0.91 


2.4 


10.26 




South 


36.1 


0.52 


36.1 


1.20 


0.0 


0.00 




West 


25.8 


0.45 


24.4 


1.03 


1.4 


5.43 





*p <.05. S ignificantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 
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Table H.30. Comparisons of respondents and nonrespondents to student questionnaire item Q12e (at 

home- grandparents) 

Respondents Nonrespondents 





(n=3769) 




(n=1464) 












Standard 


Standard 




Relative 


ttest 


Variable 


Estimate 


error 


Estimate 


error 


Bias 


bias 


p value 


Total number of students in 
















school 


1467.1 


16.17 


1416.7 


25.5 


50.4 


3.44 


0.106 


Estimated number of age- 
eligible students in school 


371.7 


4.44 


357.8 


7.16 


13.9 


3.74 


0.114 


Percentage free or reduced- 
price lunch students 


34.4 

(n=3516) 


0.41 


35.3 

(n=1372) 


0.70 


-0.9 


-2.62 


0.266 



SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.31. Types of students for respondents and nonrespondents to student questionnaire item 
Q12f (at home- other) 



— 


Respondents 

(n=3762) 

Standard 
Percent error 


Nonrespondents 

(n=1471) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


Student gender 














0.545 


Female 


48.4 


0.87 


49.4 


1.40 


-1.0 


-2.07 




Male 


51.6 


0.87 


50.6 


1.40 


1.0 


1.94 




Student grade level 














0.757 


9 or Lower 


11.0 


0.54 


11.3 


0.85 


-0.3 


-2.73 




10 or Higher 


89.0 


0.54 


88.7 


0.85 


0.3 


0.34 




Hispanic or Latino (Respondents n=3744, Nonrespondents n=1426) 






0.504 


Yes 


23.0 


0.70 


23.9 


1.18 


-0.9 


-3.91 




No 


77.0 


0.70 


76.1 


1.18 


0.9 


1.17 




White (Nonrespondents n=1455) 












0.015* 


Yes 


68.1 


0.79 


64.3 


1.32 


3.8 


5.58 




No 


31.9 


0.79 


35.7 


1.32 


-3.8 


-11.91 




Black or African American (Nonrespondents n=1455) 








0.007* 


Yes 


14.5 


0.61 


17.7 


1.02 


-3.2 


-22.07 




No 


85.5 


0.61 


82.3 


1.02 


3.2 


3.74 




Asian (Nonrespondents n=1455) 












0.015* 


Yes 


5.6 


0.39 


4.0 


0.53 


1.6 


28.57 




No 


94.4 


0.39 


96.0 


0.53 


-1.6 


-1.69 




American Indian or Alaska Native (Nonrespondents n=1455) 








0.353 


Yes 


5.5 


0.39 


4.9 


0.58 


0.6 


10.91 




No 


94.5 


0.39 


95.1 


0.58 


-0.6 


-0.63 




Native Hawaiian or Other Pacific Islander (Nonrespondents n=1455) 






0.602 


Yes 


2.8 


0.27 


3.1 


0.46 


-0.3 


-10.71 




No 


97.2 


0.27 


96.9 


0.46 


0.3 


0.31 




School control 














0.432 


Public 


91.5 


0.44 


90.4 


0.99 


1.1 


1.20 




P rivate 


8.5 


0.44 


9.6 


0.99 


-1.1 


-12.94 




Community type 














0.173 


Central city 


29.0 


0.77 


30.5 


1.30 


-1.5 


-5.17 




Urban area 


35.1 


0.83 


32.2 


1.34 


2.9 


8.26 




Town 


14.3 


0.58 


16.0 


0.97 


-1.7 


-11.89 




Rural area 


21.7 


0.67 


21.3 


1.06 


0.4 


1.84 




Region 














0.009* 


Northeast 


14.7 


0.38 


18.2 


0.91 


-3.5 


-23.81 




Midwest 


23.2 


0.41 


21.7 


0.92 


1.5 


6.47 




South 


35.9 


0.52 


36.8 


1.20 


-0.9 


-2.51 




West 


26.3 


0.44 


23.2 


1.01 


3.1 


11.79 





*p <.05. S ignificantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 
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Table H.32. Comparisons of respondents and nonrespondents to student questionnaire item Q12f 

(at home- other) 

Respondents Nonrespondents 





(n=3762) 




(n=1471) 












Standard 


Standard 




Relative 


ttest 


Variable 


Estimate 


error 


Estimate 


error 


Bias 


bias 


p value 


Total number of students in 
















school 


1464,8 


16,14 


1422,6 


25,64 


42.2 


2.88 


0,178 


Estimated number of age- 
eligible students in school 


371,2 


4,43 


359,1 


7,20 


12,1 


3,26 


0,169 


Percentage free or reduced- 
price lunch students 


34,3 

(n=3515) 


0,41 


35.5 

(n=1373) 


0,70 


-1.2 


-3.50 


0,141 



SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009, 
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Table H.33. Types of students for respondents and nonrespondents to student questionnaire 
item Q15a (mother's education- master's, doctoral or professional degree) 



— 


Respondents 

(n=4348) 

Standard 
Percent error 


Nonrespondents 

(n=885) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


Student gender 














0.001* 


Female 


49.8 


0.81 


43.3 


1.76 


6.5 


13.05 




Male 


50.2 


0.81 


56.7 


1.76 


-6.5 


-12.95 




Student grade level 














0.017* 


9 or Lower 


10.5 


0.49 


13.6 


1.21 


-3.1 


-29.52 




10 or Higher 


89.5 


0.49 


86.4 


1.21 


3.1 


3.46 




Hispanic or Latino (Respondents n=4329, Nonrespondents n=841) 








0.137 


Yes 


23.6 


0.66 


21.2 


1.48 


2.4 


10.17 




No 


76.4 


0.66 


78.8 


1.48 


-2.4 


-3.14 




White (Nonrespondents n =869) 












0.006* 


Yes 


67.9 


0.73 


62.7 


1.72 


5.2 


7.66 




No 


32.1 


0.73 


37.3 


1.72 


-5.2 


-16.20 




Black or African American (Nonrespondents n =869) 










0.001* 


Yes 


14.6 


0.55 


19.5 


1.40 


-4.9 


-33.56 




No 


85.4 


0.55 


80.5 


1.40 


4.9 


5.74 




Asian (Nonrespondents n =869) 












0.008* 


Yes 


4.7 


0.33 


7.3 


0.93 


-2.6 


-55.32 




No 


95.3 


0.33 


92.7 


0.93 


2.6 


2.73 




American Indian or Alaska Native (Nonrespondents n=869) 








0.000* 


Yes 


5.9 


0.37 


2.9 


0.57 


3.0 


50.85 




No 


94.2 


0.37 


97.1 


0.57 


-2.9 


-3.08 




Native Hawaiian or Other Pacific Islander (Nonrespondents n=869) 








0.228 


Yes 


3.0 


0.27 


2.3 


0.50 


0.7 


23.33 




No 


97.0 


0.27 


97.7 


0.50 


-0.7 


-0.72 




School control 














0.176 


Public 


90.9 


0.37 


92.7 


1.11 


-1.8 


-1.98 




P rivate 


9.1 


0.37 


7.3 


1.11 


1.8 


19.78 




Community type 














0.675 


Central city 


29.2 


0.72 


30.3 


1.64 


-1.1 


-3.77 




Urban area 


34.1 


0.77 


35.0 


1.72 


-0.9 


-2.64 




Town 


14.8 


0.54 


14.7 


1.20 


0.1 


0.68 




Rural area 


21.9 


0.62 


20.0 


1.37 


1.9 


8.68 




Region 














0.002* 


Northeast 


14.9 


0.29 


19.9 


1.30 


-5.0 


-33.56 




Midwest 


22.5 


0.31 


23.8 


1.31 


-1.3 


-5.78 




South 


36.7 


0.41 


33.3 


1.57 


3.4 


9.26 




West 


25.9 


0.34 


23.0 


1.41 


2.9 


11.20 





*p <.05. S ignificantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.34. Comparisons of respondents and nonrespondents to student questionnaire item Q15a 
(mother's education- master's, doctoral or professional degree) 

Respondents Nonrespondents 

(n=4348) (n=885) 



Standard Standard Relative ttest 



Variable 


Estimate 


error 


Estimate 


error 


Bias 


bias 


p value 


Total number of students in 


school 


1452.4 


14.71 


1455.2 


33.51 


-2.8 


-0.19 


0.939 


Estimated number of age- 


eligible students in school 


367.6 


4.04 


368.6 


9.18 


-1.0 


-0.27 


0.924 


Percentage free or reduced- 
price lunch students 


34.9 

(n=4053) 


0.38 


33.6 

(n=835) 


0.89 


1.3 


3.72 


0.191 



SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.35. Types of students for respondents and nonrespondents to student questionnaire 
item Q15c (mother's education- associate's degree) 



— 


Respondents 

(n=4369) 

Standard 
Percent error 


Nonrespondents 

(n=864) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


Student gender 














0.000* 


Female 


50.1 


0.81 


41.7 


1.83 


8.4 


16.77 




Male 


49.9 


0.81 


58.3 


1.83 


-8.4 


-16.83 




Student grade level 














0.005* 


9 or Lower 


10.4 


0.49 


14.2 


1.23 


-3.8 


-36.54 




10 or Higher 


89.6 


0.49 


85.8 


1.23 


3.8 


4.24 




Hispanic or Latino (Respondents n=4350, Nonrespondents n =820) 








0.004* 


Yes 


24.0 


0.66 


19.4 


1.45 


4.6 


19.17 




No 


76.0 


0.66 


80.6 


1.45 


-4.6 


-6.05 




White (Nonrespondents n =848) 












0.003* 


Yes 


68.0 


0.72 


62.1 


1.80 


5.9 


8.68 




No 


32.0 


0.72 


37.9 


1.80 


-5.9 


-18.44 




Black or African American (Nonrespondents n =848) 










0.000* 


Yes 


14.3 


0.54 


21.0 


1.54 


-6.7 


-46.85 




No 


85.7 


0.54 


79.0 


1.54 


6.7 


7.82 




Asian (Nonrespondents n =848) 












0.003* 


Yes 


4.7 


0.33 


7.6 


0.94 


-2.9 


-61.70 




No 


95.3 


0.33 


92.4 


0.94 


2.9 


3.04 




American Indian or Alaska Native (Nonrespondents n=848) 








0.000* 


Yes 


5.9 


0.37 


2.5 


0.53 


3.4 


57.63 




No 


94.1 


0.37 


97.5 


0.53 


-3.4 


-3.61 




Native Hawaiian or Other Pacific Islander (Nonrespondents n=848) 








0.233 


Yes 


3.0 


0.26 


2.3 


0.51 


0.7 


23.33 




No 


97.0 


0.26 


97.7 


0.51 


-0.7 


-0.72 




School control 














0.242 


Public 


91.5 


0.37 


89.6 


1.41 


1.9 


2.08 




P rivate 


8.5 


0.37 


10.4 


1.41 


-1.9 


-22.35 




Community type 














0.036* 


Central city 


28.8 


1.71 


32.1 


1.74 


-3.3 


-11.46 




Urban area 


34.1 


0.76 


35.4 


1.78 


-1.3 


-3.81 




Town 


14.8 


0.54 


14.7 


1.21 


0.1 


0.68 




Rural area 


22.3 


0.63 


17.9 


1.32 


4.4 


19.73 




Region 














0.006* 


Northeast 


15.0 


0.29 


19.4 


1.30 


-4.4 


-29.33 




Midwest 


22.7 


0.31 


23.2 


1.33 


-0.5 


-2.20 




South 


36.3 


0.41 


35.6 


1.68 


0.7 


1.93 




West 


26.1 


0.34 


21.8 


1.38 


4.3 


16.48 





*p <.05. S ignificantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 
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Table H.36. Comparisons of respondents and nonrespondents to student questionnaire item Q15c (mother's 



education- 


associate's degree) 














Respondents 
(n =4639) 


Nonrespondents 

(n=864) 










Standard 




Standard 




Relative 


ttest 


Variable 


Estimate error 


Estimate 


error 


Bias 


bias 


p value 


Total number of students in 














school 


1454.5 14.75 


1444.8 


33.81 


9.7 


0.67 


0.797 


Estimated number of age- 
eligible students in school 


368.7 4.05 


363.3 


9.42 


5.4 


1.46 


0.607 


Percentage free or reduced- 
price lunch students 


34.9 

(n=4088) 0.38 


33.6 

(n=800) 


0.91 


1.3 


3.72 


0.196 



SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.37. Types of students for respondents and nonrespondents to student questionnaire 
item Q15d (mother's education- vocational or technical certificate) 



— 


Respondents 

(n=4232) 

Standard 
Percent error 


Nonrespondents 

(n=1001) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


Student gender 














0.000* 


Female 


50.1 


0.82 


42.9 


1.70 


7.2 


14.37 




Male 


49.9 


0.82 


57.1 


1.70 


-7.2 


-14.43 




Student grade level 














0.035* 


9 or Lower 


10.6 


0.50 


13.1 


1.11 


-2.5 


-23.58 




10 or Higher 


89.5 


0.50 


86.9 


1.11 


-0.4 


-0.45 




Hispanic or Latino (Respondents n=4214, Nonrespondents n=1001) 






0.001* 


Yes 


24.2 


0.67 


19.0 


1.32 


5.2 


21.49 




No 


75.8 


0.67 


81.0 


1.32 


-5.2 


-6.86 




White (Nonrespondents n=985) 












0.020* 


Yes 


67.8 


0.74 


63.6 


1.64 


4.2 


6.19 




No 


32.2 


0.74 


36.4 


1.64 


-4.2 


-13.04 




Black or African American (Nonrespondents n=985) 








0.000* 


Yes 


14.2 


0.54 


20.4 


1.41 


-6.2 


-43.66 




No 


85.8 


0.54 


79.6 


1.41 


6.2 


7.23 




Asian (Nonrespondents n=985) 












0.004* 


Yes 


4.7 


0.33 


7.3 


0.86 


-2.6 


-55.32 




No 


95.4 


0.33 


92.7 


0.86 


2.7 


2.83 




American Indian or Alaska Native (Nonrespondents n=985) 








0.000* 


Yes 


5.9 


0.38 


3.1 


0.55 


2.8 


47.46 




No 


94.1 


0.38 


96.9 


0.55 


-2.8 


-2.98 




Native Hawaiian or Other Pacific Islander (Nonrespondents n=985) 








0.404 


Yes 


3.0 


0.27 


2.5 


0.50 


0.5 


16.67 




No 


97.0 


0.27 


97.5 


0.50 


-0.5 


-0.52 




School control 














0.436 


Public 


91.4 


0.38 


90.2 


1.28 


1.2 


1.31 




P rivate 


8.6 


0.38 


9.8 


1.28 


-1.2 


-13.95 




Community type 














0.072 


Central city 


28.8 


0.72 


31.9 


1.62 


-3.1 


-10.76 




Urban area 


34.2 


0.78 


34.7 


1.64 


-0.5 


-1.46 




Town 


14.8 


0.55 


14.7 


1.13 


0.1 


0.68 




Rural area 


22.3 


0.64 


18.7 


1.23 


3.6 


16.14 




Region 














0.001* 


Northeast 


14.9 


0.31 


19.2 


1.20 


-4.3 


-28.86 




Midwest 


22.5 


0.34 


23.9 


1.22 


-1.4 


-6.22 




South 


36.3 


0.43 


35.6 


1.54 


0.7 


1.93 




West 


26.4 


0.36 


21.3 


1.26 


5.1 


19.32 





*p <.05. S ignificantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 
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Table H.38. Comparisons of respondents and nonrespondents to student questionnaire item Q15d 
(mother's education- vocational or technical certificate) 

Respondents Nonrespondents 





(n=4232) 




(n=1001) 












Standard 


Standard 




Relative 


ttest 


Variable 


Estimate 


error 


Estimate 


error 


Bias 


Bias 


p value 


Total number of students in 
















school 


1452.8 


15.04 


1453.4 


31.32 


-0.6 


-0.04 


0.986 


Estimated number of age- 
eligible students in school 


368.2 


4.13 


366.3 


8.72 


1.9 


0.52 


0.850 


Percentage free or reduced- 
price lunch students 


35.1 

(n=3955) 


0.39 


32.8 

(n=933) 


0.83 


2.3 


6.55 


0.013* 



*p <.05. Significantly different at the .05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.39. Types of students for respondents and nonrespondents to student questionnaire 
item Q18a (father's education- Master's, doctoral or professional degree) 



- 


Respondents 

(n=4074) 

Standard 
Percent error 


Nonrespondents 
(n =1159) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


Student gender 














0.013* 


Female 


49.7 


0.84 


45.3 


1.56 


4.4 


8.85 




Male 


50.3 


0.84 


54.8 


1.56 


-4.5 


-8.95 




Student grade level 














0.005* 


9 or Lower 


10.3 


0.50 


13.6 


1.05 


-3.3 


-32.04 




10 or Higher 


89.7 


0.50 


86.4 


1.05 


3.3 


3.68 




Hispanic or Latino (Respondents n=4059, Nonrespondents n=llll) 








0.461 


Yes 


23.5 


0.68 


22.4 


1.30 


1.1 


4.68 




No 


76.5 


0.68 


77.6 


1.30 


-1.1 


-1.44 




White (Nonrespondents n=1143) 












0.000* 


Yes 


68.9 


0.74 


60.3 


1.54 


8.6 


12.48 




No 


31.1 


0.74 


39.7 


1.54 


-8.6 


-27.65 




Black or African American (Nonrespondents n=1143) 








0.000* 


Yes 


13.5 


0.54 


22.0 


1.32 


-8.5 


-62.96 




No 


86.5 


0.54 


78.0 


1.32 


8.5 


9.83 




Asian (Nonrespondents n=1143) 












0.136 


Yes 


4.9 


0.35 


6.1 


0.73 


-1.2 


-24.49 




No 


95.1 


0.35 


93.9 


0.73 


1.2 


1.26 




American Indian or Alaska Native (Nonrespondents n=1143) 








0.000* 


Yes 


6.0 


0.39 


3.0 


0.52 


3.0 


50.00 




No 


94.0 


0.39 


97.0 


0.52 


-3.0 


-3.19 




Native Hawaiian or Other Pacific Islander (Nonrespondents n=1143) 








0.158 


Yes 


3.0 


0.28 


2.3 


0.44 


0.7 


23.33 




No 


97.0 


0.28 


97.7 


0.44 


-0.7 


-0.72 




School control 














0.070 


Public 


90.7 


0.39 


93.0 


1.06 


-2.3 


-2.54 




P rivate 


9.4 


0.39 


7.0 


1.06 


2.4 


25.53 




Community type 














0.055 


Central city 


28.8 


0.73 


31.4 


1.48 


-2.6 


-9.03 




Urban area 


34.0 


0.79 


35.2 


1.52 


-1.2 


-3.53 




Town 


14.8 


0.56 


14.6 


1.05 


0.2 


1.35 




Rural area 


22.4 


0.65 


18.7 


1.16 


3.7 


16.52 




Region 














0.000* 


Northeast 


14.6 


0.33 


19.6 


1.09 


-5.0 


-34.25 




Midwest 


22.6 


0.35 


23.4 


1.11 


-0.8 


-3.54 




South 


36.4 


0.46 


35.1 


1.39 


1.3 


3.57 




West 


26.4 


0.38 


21.9 


1.16 


4.5 


17.05 





*p <.05. S ignificantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 
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Table H.40. Comparisons of respondents and nonrespondents to student questionnaire item Q18a 

(father's education- master's, doctoral or professional degree) 

Respondents Nonrespondents 





(n =4074) 




(n=1159) 












Standard 


Standard 




Relative 


ttest 


Variable 


Estimate 


error 


Estimate 


error 


Bias 


bias 


p value 


Total number of students in 
















school 


1437.1 


15.10 


1507.1 


30.61 


-70 


-4.87 


0.045* 


Estimated number of age- 
eligible students in school 


363.7 


4.15 


382.0 


8.48 


-18.3 


-5.03 


0.059 


Percentage free or reduced- 
price lunch students 


34.7 

(n=3784) 


0.39 


34.6 

(n=1104) 


0.77 


0.1 


0.29 


0.862 



*p <.05. Significantly different at the .05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.41. Types of students for respondents and nonrespondents to student questionnaire 
item Q18b (father's education- bachelor's degree) 



— 


Respondents 

(n=4232) 

Standard 
Percent error 


Nonrespondents 

(n=1001) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


Student gender 














0.023* 


Female 


49.5 


0.82 


45.2 


1.70 


4.3 


8.69 




Male 


50.5 


0.82 


54.8 


1.70 


-4.3 


-8.51 




Student grade level 














0.000* 


9 or Lower 


9.9 


0.48 


15.8 


1.22 


-5.9 


-59.60 




10 or Higher 


90.1 


0.48 


84.2 


1.22 


5.9 


6.55 




Hispanic or Latino (Respondents n=4218, Nonrespondents n=952) 








0.324 


Yes 


22.9 


0.66 


24.5 


1.47 


-1.6 


-6.99 




No 


77.1 


0.66 


75.5 


1.47 


1.6 


2.08 




White (Nonrespondents n=985) 












0.000* 


Yes 


69.4 


0.72 


57.0 


1.69 


12.4 


17.87 




No 


30.6 


0.72 


43.1 


1.69 


-12.5 


-40.85 




Black or African American (Nonrespondents n=985) 








0.000* 


Yes 


13.6 


0.54 


23.2 


1.46 


-9.6 


-70.59 




No 


86.4 


0.54 


76.8 


1.46 


9.6 


11.11 




Asian (Nonrespondents n=985) 












0.665 


Yes 


5.1 


0.35 


5.5 


0.74 


-0.4 


-7.84 




No 


94.9 


0.35 


94.6 


0.74 


0.3 


0.32 




American Indian or Alaska Native (Nonrespondents n=985) 








0.000* 


Yes 


5.9 


0.38 


3.2 


0.57 


2.7 


45.76 




No 


94.1 


0.38 


96.8 


0.57 


-2.7 


-2.87 




Native Hawaiian or Other Pacific Islander (Nonrespondents n=985) 








0.708 


Yes 


2.9 


0.26 


2.7 


0.52 


0.2 


6.90 




No 


97.1 


0.26 


97.3 


0.52 


-0.2 


-0.21 




School control 














0.061 


Public 


90.7 


0.36 


93.2 


1.18 


-2.5 


-2.76 




P rivate 


9.3 


0.36 


6.8 


1.18 


2.5 


26.88 




Community type 














0.010* 


Central city 


28.4 


0.71 


33.5 


1.63 


-5.1 


-17.96 




Urban area 


34.7 


0.78 


32.3 


1.61 


2.4 


6.92 




Town 


14.7 


0.55 


15.1 


1.14 


-0.4 


-2.72 




Rural area 


22.2 


0.63 


19.0 


1.27 


3.2 


14.41 




Region 














0.004* 


Northeast 


14.9 


0.31 


19.0 


1.19 


-4.1 


-27.52 




Midwest 


22.3 


0.33 


24.4 


1.25 


-2.1 


-9.42 




South 


36.6 


0.42 


34.1 


1.53 


2.5 


6.83 




West 


26.1 


0.36 


22.5 


1.28 


3.6 


13.79 





*p <.05. S ignificantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.42. Comparisons of respondents and nonrespondents to student questionnaire item Q18b 
(father's education- bachelor's degree) 

Respondents Nonrespondents 





(n=4232) 




(n=1001) 












Standard 


Standard 




Relative 


ttest 


Variable 


Estimate 


error 


Estimate 


error 


Bias 


bias 


p value 


Total number of students in 
















school 


1440.9 


14.68 


1502.8 


33.82 


-61.9 


-4.30 


0.100 


Estimated number of age- 
eligible students in school 


364.8 


4.03 


380.0 


9.35 


-15.2 


-4.17 


0.146 


Percentage free or reduced- 
price lunch students 


34.1 

(n=3934) 


0.38 


37.0 

(n=954) 


0.84 


-2.9 


-8.50 


0.002* 



*p <.05. Significantly different at the .05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 
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Table H.43. Types of students for respondents and nonrespondents to student questionnaire 
item Q18c (father's education-associate's degree) 



— 


Respondents 

(n=4052) 

Standard 
Percent error 


Nonrespondents 

(n=1181) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


Student gender 














0.000* 


Female 


50.2 


0.84 


43.6 


1.56 


6.6 


13.15 




Male 


49.8 


0.84 


56.4 


1.56 


-6.6 


-13.25 




Student grade level 














0.003* 


9 or Lower 


10.3 


0.50 


13.7 


1.05 


-3.4 


-33.01 




10 or Higher 


89.7 


0.50 


86.3 


1.05 


3.4 


3.79 




Hispanic or Latino (Respondents n=4037, Nonrespondents n=1133) 






0.343 


Yes 


23.5 


0.68 


22.1 


1.29 


1.4 


5.96 




No 


76.5 


0.68 


77.9 


1.29 


-1.4 


-1.83 




White (Nonrespondents n=1165) 












0.000* 


Yes 


68.8 


0.74 


60.8 


1.52 


8.0 


11.63 




No 


31.2 


0.74 


39.2 


1.52 


-8.0 


-25.64 




Black or African American (Nonrespondents n=1165) 








0.000* 


Yes 


13.8 


0.55 


20.9 


1.29 


-7.1 


-51.45 




No 


86.2 


0.55 


79.1 


1.29 


7.1 


8.24 




Asian (Nonrespondents n=1165) 












0.009* 


Yes 


4.7 


0.34 


6.8 


0.76 


-2.1 


-44.68 




No 


95.3 


0.34 


93.2 


0.76 


2.1 


2.20 




American Indian or Alaska Native (Nonrespondents n=1165) 








0.000* 


Yes 


5.9 


0.39 


3.4 


0.54 


2.5 


42.37 




No 


94.1 


0.39 


96.6 


0.54 


-2.5 


-2.66 




Native Hawaiian or Other Pacific Islander (Nonrespondents n=1165) 






0.243 


Yes 


3.0 


0.28 


2.4 


0.45 


0.6 


20.00 




No 


97.0 


0.28 


97.6 


0.45 


-0.6 


-0.62 




School control 














0.714 


Public 


91.3 


0.40 


90.8 


1.10 


0.5 


0.55 




P rivate 


8.7 


0.40 


9.2 


1.10 


-0.5 


-5.75 




Community type 














0.000* 


Central city 


28.2 


0.74 


33.3 


1.48 


-5.1 


-18.09 




Urban area 


34.1 


0.80 


34.9 


1.50 


-0.8 


-2.35 




Town 


14.9 


0.57 


14.4 


1.03 


0.5 


3.36 




Rural area 


22.9 


0.66 


17.3 


1.11 


5.6 


24.45 




Region 














0.002* 


Northeast 


14.8 


0.34 


18.9 


1.06 


-4.1 


-27.70 




Midwest 


22.6 


0.36 


23.1 


1.10 


-0.5 


-2.21 




South 


36.2 


0.46 


35.9 


1.38 


0.3 


0.83 




West 


26.4 


0.39 


22.1 


1.15 


4.3 


16.29 





*p <.05. Significantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.44. Comparisons of respondents and nonrespondents to student questionnaire item Q18c (father's 
education- associate's degree) 

Respondents Nonrespondents 





(n=4052) 




(n=1181) 












Standard 


Standard 




Relative 


ttest 


Variable 


Estimate 


error 


Estimate 


error 


Bias 


bias 


p value 


Total number of students in 
















school 


1432.1 


15.22 


1522.6 


30.01 


-90.5 


-6.32 


0.009* 


Estimated number of age- 
eligible students in school 


362.8 


4.19 


384.6 


8.31 


-21.8 


-6.01 


0.024* 


Percentage free or reduced- 
price lunch students 


34.9 

(n=3787) 


0.39 


33.9 

(n=1101) 


0.77 


1.0 


2.87 


0.278* 



*p <.05. Significantly different at the .05 level of statistical significance. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 
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Table H.45. Types of students for respondents and nonrespondents to student questionnaire 
item Q18d (father's education- vocational or technical certificate) 



— 


Respondents 

(n=4060) 

Standard 
Percent error 


Nonrespondents 

(n=1173) 

Standard 
Percent error 


Bias 


Relative 

bias 


Chi-square 
p value 


Student gender 














0.001* 


Female 


50.1 


0.83 


44.0 


1.58 


6.1 


12.18 




Male 


49.9 


0.83 


56.1 


1.58 


-6.2 


-12.42 




Student grade level 














0.018* 


9 or Lower 


10.4 


0.51 


13.1 


1.02 


-2.7 


-25.96 




10 or Higher 


89.6 


0.51 


86.9 


1.02 


2.7 


3.01 




Hispanic or Latino (Respondents n=4042, Nonrespondents n=1128) 






0.006* 


Yes 


24.1 


0.69 


20.2 


1.23 


3.9 


16.18 




No 


75.9 


0.69 


79.8 


1.23 


-3.9 


-5.14 




White (Nonrespondents n=1157) 












0.010* 


Yes 


68.0 


0.75 


63.6 


1.50 


4.4 


6.47 




No 


32.0 


0.75 


36.4 


1.50 


-4.4 


-13.75 




Black or African American (Nonrespondents n=1157) 








0.000* 


Yes 


14.2 


0.56 


19.6 


1.26 


-5.4 


-38.03 




No 


85.8 


0.56 


80.5 


1.26 


5.3 


6.18 




Asian (Nonrespondents n=1157) 












0.013* 


Yes 


4.7 


0.34 


6.8 


0.76 


-2.1 


-44.68 




No 


95.3 


0.34 


93.2 


0.76 


2.1 


2.20 




American Indian or Alaska Native (Nonrespondents n=1157) 








0.000* 


Yes 


6.0 


0.39 


3.2 


0.52 


2.8 


46.67 




No 


94.0 


0.39 


96.8 


0.52 


-2.8 


-2.98 




Native Hawaiian or Other Pacific Islander (Nonrespondents n=1157) 






0.563 


Yes 


2.9 


0.27 


2.6 


0.47 


0.3 


10.34 




No 


97.1 


0.27 


97.4 


0.47 


-0.3 


-0.31 




School control 














0.184 


Public 


91.6 


0.40 


89.7 


1.19 


1.9 


2.07 




P rivate 


8.4 


0.40 


10.3 


1.19 


-1.9 


-22.62 




Community type 














0.000* 


Central city 


28.6 


0.74 


31.9 


1.48 


-3.3 


-11.54 




Urban area 


33.5 


0.78 


36.8 


1.55 


-3.3 


-9.85 




Town 


15.0 


0.57 


13.8 


1.01 


1.2 


8.00 




Rural area 


22.8 


0.66 


17.4 


1.09 


5.4 


23.68 




Region 














0.000* 


Northeast 


14.6 


0.33 


19.5 


1.08 


-4.9 


-33.56 




Midwest 


22.4 


0.36 


23.9 


1.12 


-1.5 


-6.70 




South 


36.1 


0.46 


36.1 


1.40 


0.0 


0.00 




West 


26.9 


0.39 


20.5 


1.11 


6.4 


23.79 





*p <.05. S ignificantly different at the .05 level of statistical significance. 

NOTE: Detail may notsum to totals due to rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2009. 
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Table H.46. Comparisons of respondents and nonrespondents to student questionnaire item Q18d (father's 
education- vocational or technical certificate) 

Respondents Nonrespondents 





(n =4060) 




(n=1173) 












Standard 


Standard 




Relative 


ttest 


Variable 


Estimate 


error 


Estimate 


error 


Bias 


bias 


p value 


Total number of students in 
















school 


1443.9 


15.4 


1483.1 


29.31 


-39.2 


-2.71 


0.249 


Estimated number of age- 
eligible students in school 


365.9 


4.23 


374.2 


8.19 


-8.3 


-2.27 


0.381 


Percentage free or reduced- 
price lunch students 


34.9 

(n=3799) 


0.39 


33.8 

(n=1089) 


0.76 


1.1 


3.15 


0.198 



SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (P ISA), 2009. 



2009 Data Files and Database with U.S. Specific Variables 



H-41 




